Multimedia data with associated semantics is omnipresent in today’s social online platforms in the form of keywords, user comments and so forth. This article presents a statistical framework designed to infer knowledge in the imaging domain from the semantic domain. Note that this is the reverse direction of common computer vision applications. The framework relates keywords to image characteristics using a statistical significance test. It scales to millions of images and hundreds of thousands of keywords. We demonstrate the usefulness of the statistical framework with three color imaging applications. 1) semantic image enhancement: re-render an image in order to adapt it to its semantic context 2) color naming: find the color triplet for a given color name 3) color palettes: find a palette of colors that best represents a given arbitrary semantic context and that satisfies established harmony constraints