Using functional magnetic resonance imaging, we estimated neural activity in twins to study genetic influences on the cortical response to categories of visual stimuli (faces, places, and pseudowords) that are known to elicit distinct patterns of activity in ventral visual cortex. The neural activity patterns in monozygotic twins were significantly more similar than in dizygotic twins for the face and place stimuli, but there was no effect of zygosity for pseudowords (or chairs, a control category). These results demonstrate that genetics play a significant role in determining the cortical response to faces and places, but play a significantly smaller role (if any) in the response to orthographic stimuli.