We perform two-dimensional coherent spectroscopy on CdSe/CdZnS core-shell colloidal quantum dots at cryogenic temperatures. In the two-dimensional spectra, sidebands due to electronic coupling with CdSe lattice LO-phonon modes are observed to have evolutions deviating from the exponential dephasing expected from Markovian spectral diffusion, which is instantaneous and memoryless. Comparison to simulations provides evidence that LO-phonon coupling induces energy-gap fluctuations on the finite timescales of nuclear motion. The femtosecond resolution of our technique probes exciton dynamics directly on the timescales of phonon coupling in nanocrystals.