Cancer cells-derived extracellular vesicles can trigger the transformation of adipose-derived mesenchymal stem cells (ADMSC) into a pro-inflammatory, cancer-associated adipocyte (CAA) phenotype. Such secretome-mediated crosstalk between the adipose tissue and the tumor microenvironment (TME) therefore impacts tumor progression and metastatic processes. In addition, emerging roles of diet-derived phytochemicals, especially epigallocatechin-3-gallate (EGCG) among other polyphenols, in modulating exosome-mediated metabolic and inflammatory signaling pathways have been highlighted. Here, we discuss how selected diet-derived phytochemicals could alter the secretome signature as well as the crosstalk dynamics between the adipose tissue and the TME, with a focus on breast cancer. Their broader implication in the chemoprevention of obesity-related cancers is also discussed.