Phytoestrogens are increasingly used as dietary supplements due to their suggested health promoting properties, but also by women for breast enhancement and relief of menopausal symptoms. Generally, phytoestrogens are considered to exert estrogenic activity via estrogen receptors (ERs), but they may also affect estrogen synthesis and metabolism locally in breast, endometrial and ovarian tissues. Considering that accurate regulation of local hormone levels is crucial for normal physiology, it is not surprising that interference with hormonal synthesis and metabolism is associated with a wide variety of women's health problems, varying from altered menstrual cycle to hormone-dependent cancers. Yet, studies on phytoestrogens have mainly focused on ER-mediated effects of soy-derived phytoestrogens, with less attention paid to steroid synthesis and metabolism or other phytoestrogens. This review aims to evaluate the potential of phytoestrogens to modulate local estrogen levels and the implications for women's health. For that, an overview is provided of the effects of commonly used phytoestrogens, i.e. 8-prenylnaringenin, biochanin A, daidzein, genistein, naringenin, resveratrol and quercetin, on estrogen synthesizing and metabolizing enzymes in vitro. The potential implications for women's health are assessed by comparing the in vitro effect concentrations with blood concentrations that can be found after intake of these phytoestrogens. Based on this evaluation, it can be concluded that high-dose supplements with phytoestrogens might affect breast and endometrial health or fertility in women via the modulation of steroid hormone levels. However, more data regarding the tissue levels of phytoestrogens and effect data from dedicated, tissue-specific assays are needed for a better understanding of potential risks. At least until more certainty regarding the safety has been established, especially young women would better avoid using supplements containing high doses of phytoestrogens.