Age-related deficits in episodic retrieval have been associated with volume reductions in the middle frontal gyrus (MFG). However, it remains unclear how this age-related reduction in MFG volume correlates with neural activity during retrieval. To address this, we conducted in vivo volumetry of the frontal cortex in young and older human adults and found more volume loss on the right than on the left MFG with age. We then examined how left and right MFG volume correlated with fMRI activity during successful retrieval of item, spatial context, and temporal context information in both age groups. In young adults, larger right MFG volume was positively correlated with greater activity in a commonly found episodic retrieval network that included bilateral dorsolateral prefrontal cortex (DLPFC) and bilateral inferior parietal cortex. Within this network, left DLPFC and right inferior parietal cortex activity predicted memory performance. In older adults, a positive structure-function association in DLPFC for either left or right MFG/DLPFC was not observed. Instead, right MFG volume was negatively correlated with activity in several regions in older adults, including the parahippocampal cortex (PHC) and anterior cingulate. Less activity in the PHC region predicted better item memory, and less activity in the anterior cingulate predicted better spatial context accuracy in older adults. We conclude that age-related change in the structure-function association in MFG/DLPFC impacts retrieval activity and performance, and those older adults with larger right MFG volume attempt to compensate for this change by modifying activity in other brain regions to help retrieval performance.