Increasing age is associated with deficits in a wide range of cognitive domains as well as with structural brain changes. Recent studies using diffusion tensor imaging (DTI) have shown that microstructural integrity of white matter is associated with cognitive performance in elderly persons, especially on tests that rely on perceptual speed. We used structural equation modeling to investigate associations between white matter microstructure and cognitive functions in a population-based sample of elderly persons (age ≥ 60 years), free of dementia, stroke, and neurological disorders (n = 253). Participants underwent a magnetic resonance imaging scan, from which mean fractional anisotropy (FA) and mean diffusivity (MD) of seven white matter tracts were quantified. Cognitive functioning was analyzed according to performance in five task domains (perceptual speed, episodic memory, semantic memory, letter fluency, and category fluency). After controlling for age, FA and MD were exclusively related to perceptual speed. When further stratifying the sample into two age groups, the associations were reliable in the old-old (≥ 78 years) only. This relationship between white matter microstructure and perceptual speed remained significant after excluding persons in a preclinical dementia phase. The observed pattern of results suggests that microstructural white matter integrity may be especially important to perceptual speed among very old adults.