The tumor microenvironment (TME) is a complex ecosystem of diverse cell types whose interactions govern tumor growth and clinical outcome. While the TME's impact on immunotherapy has been extensively studied, its role in chemotherapy response remains less explored. To address this, we developed DECODEM (DEcoupling Cell-type-specific Outcomes using DEconvolution and Machine learning), a generic computational framework leveraging cellular deconvolution of bulk transcriptomics to associate the gene expression of individual cell types in the TME with clinical response. Employing DECODEM to analyze the gene expression of breast cancer (BC) patients treated with neoadjuvant chemotherapy, we find that the gene expression of specific immune cells (myeloid, plasmablasts, B-cells) and stromal cells (endothelial, normal epithelial, CAFs) are highly predictive of chemotherapy response, going beyond that of the malignant cells. These findings are further tested and validated in a single-cell cohort of triple negative breast cancer. To investigate the possible role of immune cell-cell interactions (CCIs) in mediating chemotherapy response, we extended DECODEM to DECODEMi to identify such CCIs, validated in single-cell data. Our findings highlight the importance of active pre-treatment immune infiltration for chemotherapy success. The tools developed here are made publicly available and are applicable for studying the role of the TME in mediating response from readily available bulk tumor expression in a wide range of cancer treatments and indications.