We report on measurements of the photoluminescence properties of single nitrogen-vacancy centers in diamond at temperatures between 4 K and 300 K. We observe a strong reduction of the PL intensity and spin contrast between ca. 10 K and 100 K that recovers to high levels below and above. Further, we find a rich dependence on magnetic bias field and crystal strain. We develop a comprehensive model based on spin mixing and orbital hopping in the electronic excited state that quantitatively explains the observations. Beyond a more complete understanding of the excited-state dynamics, our work provides a novel approach for probing electron-phonon interactions and a predictive tool for optimizing experimental conditions for quantum applications.