We present the effects of resonator birefringence on the cavity-enhanced interfacing of quantum states of light and matter, including the first observation of single photons with a time-dependent polarization state that evolves within their coherence time. A theoretical model is introduced and experimentally verified by the modified polarization of temporally long single photons emitted from a ^{87}Rb atom coupled to a high-finesse optical cavity by a vacuum-stimulated Raman adiabatic passage process. Further theoretical investigation shows how a change in cavity birefringence can both impact the atom-cavity coupling and engender starkly different polarization behavior in the emitted photons. With polarization a key resource for encoding quantum states of light and modern micron-scale cavities particularly prone to birefringence, the consideration of these effects is vital to the faithful realization of efficient and coherent emitter-photon interfaces for distributed quantum networking and communications.