Trastuzumab deruxtecan (T-DXd), an anti-HER2 antibody-drug conjugate with a topoisomerase I inhibitor connected by a cleavable linker, has been approved for patients with HER2-positive gastric or gastroesophageal junction tumors. This biomarker study assessed HER2 expression and immune cell infiltration in relation to the therapeutic response to T-DXd. This retrospective analysis included samples from patients treated with T-DXd in three clinical trials. We performed RNA sequencing and multiplex immunohistochemistry on archival tumor samples obtained at baseline, during treatment, and after treatment. Flow cytometry was performed on tumor infiltrating immune cells freshly isolated from tumor tissues. Samples from 28 patients were included in this study. ERBB2 mRNA levels and CD20+ cell infiltration in tumors were significantly higher at baseline in responders than in nonresponders. Patients were classified into three biological groups based on their baseline tumor/stroma-infiltrating immune cell densities. Two groups reported similar response rates, but a trend was observed toward a shorter progression-free-survival in the group with more immunosuppressive regulatory T cells and programmed death-ligand 1 (PD-L1) expression at baseline. T-DXd treatment tended to increase the levels of tumor-infiltrating CD8+ T cells and PD1+CD8+ T cells, particularly in responders. Gene expression signatures of CTL and helper T cells increased during treatment, whereas signatures related to hypoxia, MYC targets, collagen formation, and interleukin-10 were downregulated. Our data suggest that HER2 expression levels and baseline tumor microenvironment (TME) characteristics correlate with T-DXd efficacy. Furthermore, this treatment may modulate TME immune profiles. Further validation using a larger sample size is warranted.