Previous behavioral research suggests that although elderly adults' memory benefits from supportive context, misleading or irrelevant contexts produce greater interference. In the present study, we use event-related fMRI to investigate age differences when processing contextual information to make recognition judgments. Twenty-one young and twenty elderly incidentally encoded pictures of objects presented in meaningful contexts, and completed a memory test for the objects presented in identical or novel contexts. Elderly committed more false alarms than young when novel objects were presented in familiar, but task-irrelevant, contexts. Elderly showed reduced engagement of bilateral dorsolateral prefrontal cortex and anterior cingulate relative to young, reflecting disruption of a cognitive control network for processing context with age. Disruption occurred for both high and low-performing elderly, suggesting that cognitive control deficits are pervasive with age. Despite showing disruption of the cognitive control network, high-performing elderly recruited additional middle and medial frontal regions that were not recruited by either low-performing elderly or young adults. This suggests that high-performing elderly may compensate for the disruption of the cognitive control network by recruiting additional frontal resources to overcome cognitive control deficits that affect recognition memory.