Above-threshold ionization spectra from cesium are measured as a function of the carrier-envelope phase (CEP) using laser pulses centered at 3.1 μm wavelength. The directional asymmetry in the energy spectra of backscattered electrons oscillates three times, rather than once, as the CEP is changed from 0 to 2π. Using the improved strong-field approximation, we show that the unusual behavior arises from the interference of few quantum orbits. We discuss the conditions for observing the high-order CEP dependence, and draw an analogy with time-domain holography with electron wave packets.