There is increasing recognition that set-shifting, a form of cognitive control, is mediated by different neural structures. However, these regions have not yet been carefully identified as many studies do not account for the influence of component processes (e.g., motor speed). We investigated gray matter correlates of set-shifting while controlling for component processes. Using the Design Fluency (DF), Trail Making Test (TMT), and Color Word Interference (CWI) subtests from the Delis-Kaplan Executive Function System (D-KEFS), we investigated the correlation between set-shifting performance and gray matter volume in 160 subjects with neurodegenerative disease, mild cognitive impairment, and healthy older adults using voxel-based morphometry. All three set-shifting tasks correlated with multiple, widespread gray matter regions. After controlling for the component processes, set-shifting performance correlated with focal regions in prefrontal and posterior parietal cortices. We also identified bilateral prefrontal cortex and the right posterior parietal lobe as common sites for set-shifting across the three tasks. There was a high degree of multicollinearity between the set-shifting conditions and the component processes of TMT and CWI, suggesting DF may better isolate set-shifting regions. Overall, these findings highlight the neuroanatomical correlates of set-shifting and the importance of controlling for component processes when investigating complex cognitive tasks.