Neuroimaging tests of sensorimotor theories of semantic memory hinge on the extent to which similar activation patterns are observed during perception and retrieval of objects or object properties. The present study was motivated by the hypothesis that some of the seeming discrepancies across studies reflect flexibility in the systems responsible for conceptual and perceptual processing of color. Specifically, we test the hypothesis that retrieval of color knowledge can be influenced by both context (a task variable) and individual differences in cognitive style (a subject variable). In Experiment 1, we provide fMRI evidence for differential activity during color knowledge retrieval by having subjects perform a verbal task, in which context encouraged subjects to retrieve more- or less-detailed information about the colors of named common objects in a blocked experimental design. In the left fusiform, we found more activity during retrieval of more- versus less-detailed color knowledge. We also assessed preference for verbal or visual cognitive style, finding that brain activity in the left lingual gyrus significantly correlated with preference for a visual cognitive style. We replicated many of these effects in Experiment 2, in which stimuli were presented more quickly, in a random order, and in the auditory modality. This illustration of some of the factors that can influence color knowledge retrieval leads to the conclusion that tests of conceptual and perceptual overlap must consider variation in both of these processes.