Nicotine modulates prefrontal processing when tested with functional imaging. Previous studies on changes in regional brain volumes in small samples, reporting different life-time exposure to nicotine, identified reduced volume in smokers in prefrontal areas but reported controversial results for other areas. We investigated the association of cigarette smoking and regional gray and white matter volume by using voxel-based morphometry (VBM) for T1-weighted high-resolution magnetic resonance imaging in 315 current-smokers and 659 never-smokers from the representative Study of Health in Pomerania (SHIP). Our study showed that in current-smokers smoking is significantly associated with gray matter volume loss in the prefrontal cortex, the anterior cingulate cortex, the insula, and the olfactory gyrus. White matter volumes were not relevantly reduced in current-smokers. In current-smokers, we found associations of gray matter loss and smoking exposure (pack-years) in the prefrontal cortex, the anterior and middle cingulate cortex, and the superior temporal and angular gyrus, which however did not stand corrections for multiple testing. We confirmed associations between smoking and gray matter differences in the prefrontal cortex, the anterior cingulate cortex and the insula in the general population of Pomerania (Germany). For the first time, we identified differences in brain volumes in the olfactory gyrus. Other cerebral regions did not show significant differences when correcting for multiple comparisons within the whole brain. The regions of structural deficits might be involved in addictive behavior and withdrawal symptoms, whereas further investigations have to show if the observed atrophies were caused by smoking itself or are preexisting differences between smoking and non-smoking individuals.