Brain iron increases with age and is abnormally elevated early in the disease process in several neurodegenerative disorders that impact memory including Alzheimer's disease (AD). Higher brain iron levels are associated with male gender and presence of highly prevalent allelic variants in genes encoding for iron metabolism proteins (hemochromatosis H63D (HFE H63D) and transferrin C2 (TfC2)). In this study, we examined whether in healthy older individuals memory performance is associated with increased brain iron, and whether gender and gene variant carrier (IRON+) vs noncarrier (IRON-) status (for HFE H63D/TfC2) modify the associations. Tissue iron deposited in ferritin molecules can be measured in vivo with magnetic resonance imaging utilizing the field-dependent relaxation rate increase (FDRI) method. FDRI was assessed in hippocampus, basal ganglia, and white matter, and IRON+ vs IRON- status was determined in a cohort of 63 healthy older individuals. Three cognitive domains were assessed: verbal memory (delayed recall), working memory/attention, and processing speed. Independent of gene status, worse verbal-memory performance was associated with higher hippocampal iron in men (r=-0.50, p=0.003) but not in women. Independent of gender, worse verbal working memory performance was associated with higher basal ganglia iron in IRON- group (r=-0.49, p=0.005) but not in the IRON+ group. Between-group interactions (p=0.006) were noted for both of these associations. No significant associations with white matter or processing speed were observed. The results suggest that in specific subgroups of healthy older individuals, higher accumulations of iron in vulnerable gray matter regions may adversely impact memory functions and could represent a risk factor for accelerated cognitive decline. Combining genetic and MRI biomarkers may provide opportunities to design primary prevention clinical trials that target high-risk groups.