Prevalent theories hold that semantic memory is organized by sensorimotor modality (e.g., visual knowledge, motor knowledge). While some neuroimaging studies support this idea, it cannot account for the category specific (e.g., living things) knowledge impairments seen in some brain damaged patients that cut across modalities. In this article we test an alternative model of how damage to interactive, modality-specific neural regions might give rise to these categorical impairments. Functional MRI was used to examine a cortical area with a known modality-specific function during the retrieval of visual and non-visual knowledge about living and non-living things. The specific predictions of our model regarding the signal observed in this area were confirmed, supporting the notion that semantic memory is functionally segregated into anatomically discrete, but highly interactive, modality-specific regions.