The optical transition linewidth and emission polarization of single nitrogen-vacancy (NV) centers are measured from 5 K to room temperature. Interexcited state population relaxation is shown to broaden the zero-phonon line and both the relaxation and linewidth are found to follow a T(5) dependence for T < 100 K. This dependence indicates that the dynamic Jahn-Teller effect is the dominant dephasing mechanism for the NV optical transitions at low temperatures.