We present inelastic neutron scattering experiments on a single-domain crystal of lightly doped La1.96Sr0.04CuO4. We find that the magnetic excitation spectrum in this insulating phase with a diagonal incommensurate spin modulation is remarkably similar to that in the superconducting regime, where the spin modulation is bond parallel. In particular, we find that the dispersion slope at low energy is essentially independent of doping and temperature over a significant range. The energy at which the excitations cross the commensurate antiferromagnetic wave vector increases roughly linearly with doping through the underdoped regime.