Energy restriction inhibits mammary tumor development in animal models. Epidemiologic studies in humans generally do not support an association between dietary energy intake and breast cancer risk, although some studies suggest a more complex interplay between measures of energy intake, physical activity, and body size. We examined the association between total energy intake jointly with physical activity and body mass index (BMI) and the risk of breast cancer among 1,775 women diagnosed with breast cancer between 1995 and 2006 and 2,529 of their unaffected sisters, enrolled in the Breast Cancer Family Registry. We collected dietary data using the Hawaii-Los Angeles Multiethnic Cohort food frequency questionnaire. Using conditional logistic regression to estimate the odds ratios (OR) and 95 % confidence intervals (CI) associated with total energy intake, we observed an overall 60-70 % increased risk of breast cancer among women in the highest quartile of total energy intake compared to those in the lowest quartile (Q4 vs. Q1: OR = 1.6, 95 % CI: 1.3-2.0; P (trend) < 0.0001); these associations were limited to pre-menopausal women or women with hormone receptor-positive cancers. Although the associations were slightly stronger among women with a higher BMI or lower level of average lifetime physical activity, we observed a positive association between total energy intake and breast cancer risk across different strata of physical activity and BMI. Our results suggest that within sisters, high energy intake may increase the risk of breast cancer independent of physical activity and body size. If replicated in prospective studies, then these findings suggest that reductions in total energy intake may help in modifying breast cancer risk.