Up to 25 % of patients discontinue adjuvant aromatase inhibitor (AI) therapy due to intolerable symptoms. Predictors of which patients will be unable to tolerate these medications have not been defined. We hypothesized that inherited variants in candidate genes are associated with treatment discontinuation because of AI-associated toxicity. We prospectively evaluated reasons for treatment discontinuation in women with hormone receptor-positive breast cancer initiating adjuvant AI through a multicenter, prospective, randomized clinical trial of exemestane versus letrozole. Using multiple genetic models, we evaluated potential associations between discontinuation of AI therapy because of toxicity and 138 variants in 24 candidate genes, selected a priori, primarily with roles in estrogen metabolism and signaling. To account for multiple comparisons, statistical significance was defined as p < 0.00036. Of the 467 enrolled patients with available germline DNA, 152 (33 %) discontinued AI therapy because of toxicity. Using a recessive statistical model, an intronic variant in ESR1 (rs9322336) was associated with increased risk of musculoskeletal toxicity-related exemestane discontinuation [HR 5.0 (95 % CI 2.1-11.8), p < 0.0002]. An inherited variant potentially affecting estrogen signaling may be associated with exemestane-associated toxicity, which could partially account for intra-patient differences in AI tolerability. Validation of this finding is required.