Identifying host genetic factors modulating immune checkpoint inhibitor (ICI) efficacy has been experimentally challenging because of variations in both host and tumor genomes, differences in the microbiome, and patient life exposures. Utilizing the Collaborative Cross (CC) multi-parent mouse genetic resource population, we developed an approach that fixes the tumor genomic configuration while varying host genetics. With this approach, we discovered that response to anti-PD-1 (aPD1) immunotherapy was significantly heritable in four distinct murine tumor models (H2 between 0.18-0.40). For the MC38 colorectal carcinoma system (H2 = 0.40), we mapped four significant ICI response quantitative trait loci (QTL) localized to mouse chromosomes (mChr) 5, 9, 15 and 17, and identified significant epistatic interactions between specific QTL pairs. Differentially expressed genes within these QTL were highly enriched for immune genes and pathways mediating allograft rejection and graft vs host disease. Using a cross species analytical approach, we found a core network of 48 genes within the four QTLs that showed significant prognostic value for overall survival in aPD1 treated human cohorts that outperformed all other existing validated immunotherapy biomarkers, especially in human tumors of the previously defined immune subtype 4. Functional blockade of two top candidate immune targets within the 48 gene network, GM-CSF and high affinity IL-2/IL-15 signaling, completely abrogated the MC38 tumor transcriptional response to aPD1 therapy in vivo. Thus, we have established a powerful cross species in vivo platform capable of uncovering host genetic factors that establish the tumor immune microenvironment configuration propitious for ICI response.