We use the coherent backscattering interference effect to investigate experimentally and theoretically how coherent transport of light inside a cold atomic vapor is affected by the residual motion of atomic scatterers. As the temperature of the atomic cloud increases, the interference contrast decreases dramatically. This emphasizes the role of motion-induced decoherence for resonant scatterers even in the sub-Doppler regime of temperature. We derive analytical expressions for the corresponding coherence time.