We introduce a theory to describe the interaction of swift electrons with strong evanescent light fields. This allows us to explain recent experimental results of multiple energy losses and gains for electrons passing near illuminated nanostructures. A complex evolution of the electron state over attosecond time scales is unveiled, giving rise to non-Poissonian distributions of multiphoton features in the electron spectra. Prospects for application to nanoscale-resolved transmission electron microscopy and spectroscopy are discussed.