Epidemiological studies suggest lifestyle factors may reduce the risk of dementia. However, few studies have examined the association of diet and waist-to-hip ratio with hippocampus connectivity. In the Whitehall II Imaging Sub-study, we examined longitudinal changes in diet quality in 512 participants and waist-to-hip ratio in 665 participants. Diet quality was measured using the Alternative Health Eating Index-2010 assessed three times across 11 years between ages 48 and 60 years, and waist-to-hip ratio five times over 21 years between ages 48 and 68 years. Brain imaging and cognitive tests were performed at age 70±5 years. We measured white matter using diffusion tensor imaging and hippocampal functional connectivity using resting-state functional magnetic resonance imaging. In addition to associations of diet and waist-to-hip ratio with brain imaging measures, we tested whether associations between diet, waist-to-hip ratio and cognitive performance were mediated by brain connectivity. We found better diet quality in midlife and improvements in diet over mid-to-late life were associated with higher hippocampal functional connectivity to the occipital lobe and cerebellum, and better white matter integrity as measured by higher fractional anisotropy and lower diffusivity. Higher waist-to-hip ratio in midlife was associated with higher mean and radial diffusivity and lower fractional anisotropy in several tracts including the inferior longitudinal fasciculus and cingulum. Associations between midlife waist-to-hip ratio, working memory and executive function were partially mediated by radial diffusivity. All associations were independent of age, sex, education, and physical activity. Our findings highlight the importance of maintaining a good diet and a healthy waist-to-hip ratio in midlife to maintain brain health in later life. Future interventional studies for the improvement of dietary and metabolic health should target midlife for the prevention of cognitive decline in old age.