Plasmon-enhanced light-matter interactions have been widely investigated in the past decades. Here, we report surface plasmon-enhanced structural dynamics in multi-walled carbon nanotubes. The optical polarization dependent dynamic properties of multi-walled carbon nanotubes are investigated using ultrafast transmission electron microscopy. Lattice contractions in the femtosecond time regime are observed upon excitation of the azimuthal plasmon by light polarized perpendicular to the tubular axis. The polarization dependence of the plasmon near field was examined using photon-induced near-field electron microscopy. The lattice changes resulting from the azimuthal plasmon enhance ultrafast alterations in both localized evanescent fields and the collective charge excitation, which play critical roles governing the light-matter interaction. These results suggest that the ultrafast responses of lattice degrees of freedom in nanomaterials could be essential for understanding the mechanism of surface plasmon enhanced effects.