An atomic-level picture of molecular and bulk processes, such as chemical bonding and charge transfer, necessitates an understanding of the dynamical evolution of these systems. On the ultrafast timescales associated with nuclear and electronic motion, the temporal behaviour of a system is often interrogated in a 'pump-probe' scheme. Here, an initial 'pump' pulse triggers dynamics through photoexcitation, and after a carefully controlled delay a 'probe' pulse initiates projection of the instantaneous state of the evolving system onto an informative measurable quantity, such as electron binding energy. In this paper, we apply spectral ghost imaging to a pump-probe time-resolved experiment at an X-ray free-electron laser (XFEL) facility, where the observable is spectral absorption in the X-ray regime. By exploiting the correlation present in the shot-to-shot fluctuations in the incoming X-ray pulses and measured electron kinetic energies, we show that spectral ghost imaging can be applied to time-resolved pump-probe measurements. In the experiment presented, interpretation of the measurement is simplified because spectral ghost imaging separates the overlapping contributions to the photoelectron spectrum from the pump and probe pulse.