Low intra-tumor heterogeneity (ITH) correlates with increased patient survival and immunotherapy response. However, even highly homogeneous tumors are variably aggressive, and the immunological factors impacting aggressiveness remain understudied. Here, we analyzed the mechanisms underlying immune escape in murine tumors with low ITH. We used immunophenotyping and single-cell RNA sequencing to compare the temporal growth of in-vivo transplanted, genetically similar rejected vs. non-rejected single-cell clones. Non-rejected clones showed high infiltration of tumor-associated macrophages (TAMs), lower T-cell infiltration, and increased T-cell exhaustion compared to rejected clones. Comparative analysis of rejection-associated gene expression programs, combined with in-vivo CRISPR knockout screens of candidate regulators, identified Mif (macrophage migration inhibitory factor) as a major contributor to immune rejection. Mif knockout resulted in smaller tumors and reduced TAM infiltration. These results were validated in melanoma patient data. Overall, our homogeneous tumor system can uncover factors regulating growth variability and identifies Mif as critical in aggressive melanoma.