Functional MRI during face matching shows activation of the ventral visual stream, including the ventral temporal lobes and fusiform gyrus. In contrast, a location-matching task activates the dorsal visual stream, compromising parietal lobe areas. The morphological basis of the functional coupling between brain regions may be related to the distribution of neuron numbers and neuropil density, but has not yet been demonstrated in the living human brain. Regional neuron density can indirectly be assessed in vivo using structural MRI. The progression of Alzheimer's disease pathology along specific functional systems provides an in vivo lesion model to determine the interaction between reduced neuron numbers and reduced neuronal activation. In this study, we determined correlations between activation of the fusiform gyrus in fMRI during face matching and cortical grey matter density derived from structural MRI in 17 healthy elderly subjects (mean age = 67.5 years, SD = 4.5 years, 10 women) and 16 patients with amnestic mild cognitive impairment (MCI) (mean age = 69.9 years, SD = 8.0 years, 8 women), a predementia stage of Alzheimer's disease. Independently of diagnosis, stronger activation of the fusiform gyrus was correlated with larger grey matter density in the fusiform gyrus, inferior and middle temporal gyrus, parahippocampal gyrus and dorsolateral prefrontal cortex. In contrast, smaller activation of the fusiform gyrus was associated with larger grey matter density in the inferior parietal lobule, post-central gyrus and dorsolateral prefrontal cortex. Compared to controls, MCI patients had more pronounced positive correlations in the ventral temporal lobes and more pronounced negative correlations in the parietal lobes. Our data suggest that fusiform activation is positively correlated with cortical grey matter density of brain areas belonging to the ventral visual stream and negatively correlated with grey matter density of brain areas belonging to the dorsal visual stream and that, these effects are more pronounced in MCI patients than in controls. These findings support the notion that the functional segregation within the visual system is based on the distribution of cortical grey matter volumes, possibly reflecting the spatial distribution of neuron density.