The ensemble emission spectra of colloidal InP quantum dots are broader than achievable spectra of cadmium- and lead-based quantum dots, despite similar single-particle line widths and significant efforts invested in the improvement of synthetic protocols. We seek to explain the origin of persistently broad ensemble emission spectra of colloidal InP quantum dots by investigating the nature of the electronic states responsible for luminescence. We identify a correlation between red-shifted emission spectra and anomalous broadening of the excitation spectra of luminescent InP colloids, suggesting a trap-associated emission pathway in highly emissive core-shell quantum dots. Time-resolved pump-probe experiments find that electrons are largely untrapped on photoluminescence relevant time scales pointing to emission from recombination of localized holes with free electrons. Two-dimensional electronic spectroscopy on InP quantum dots reveals multiple emissive states and increased electron-phonon coupling associated with hole localization. These localized hole states near the valence band edge are hypothesized to arise from incomplete surface passivation and structural disorder associated with lattice defects. We confirm the presence and effect of lattice disorder by X-ray absorption spectroscopy and Raman scattering measurements. Participation of localized electronic states that are associated with various classes of lattice defects gives rise to phonon-coupled defect related emission. These findings explain the origins of the persistently broad emission spectra of colloidal InP quantum dots and suggest future strategies to narrow ensemble emission lines comparable to what is observed for cadmium-based materials.