Evidence is inconsistent regarding whether dietary fat influences sex hormone concentrations. This issue is important for breast cancer survivors since clinical recommendations suggest maintaining low hormone levels primarily via pharmacologic agents. This study examines associations between dietary fat and circulating sex hormones among participants in the Health, Eating, Activity and Lifestyle (HEAL) Study, a cohort of breast cancer survivors (N = 511). During a postdiagnosis interview, detailed data were collected on diet, physical activity, lifestyle habits, and medication use (including tamoxifen). Staff measured height and weight and collected fasting bloods. Multivariate linear regression modeled associations of dietary fat with serum sex hormones. Among women using tamoxifen, we observed modest inverse associations of dietary fat with estrone (P < 0.01), estradiol (P < 0.05), testosterone (P < 0.01), free testosterone (P < 0.01), and DHEA (P < 0.01) for higher vs. lower fat intake; but there was no evidence for a trend. Associations were consistent across measures (percent energy from fat, total, saturated, and polyunsaturated fat), and modest effect modification was observed between fat intake and tamoxifen in relation to hormones. Among women not using tamoxifen, fat intake was not associated with hormone concentrations. Further work is needed to confirm the findings and to understand the clinical implications of these observations.