Interspecific hybridization can introduce genetic variation that aids in adaptation to new or changing environments. Here, we investigate how hybrid adaptation to temperature and nutrient limitation may alter parental genome representation over time. We evolved Saccharomyces cerevisiae x Saccharomyces uvarum hybrids in nutrient-limited continuous culture at 15°C for 200 generations. In comparison to previous evolution experiments at 30°C, we identified a number of responses only observed in the colder temperature regime, including the loss of the S. cerevisiae allele in favor of the cryotolerant S. uvarum allele for several portions of the hybrid genome. In particular, we discovered a genotype by environment interaction in the form of a loss of heterozygosity event on chromosome XIII; which species' haplotype is lost or maintained is dependent on the parental species' temperature preference and the temperature at which the hybrid was evolved. We show that a large contribution to this directionality is due to a temperature dependent fitness benefit at a single locus, the high affinity phosphate transporter gene PHO84. This work helps shape our understanding of what forces impact genome evolution after hybridization, and how environmental conditions may promote or disfavor the persistence of hybrids over time.