The combined effect of single-particle and collective surface excitations in the decay of image-potential states on Ag surfaces is investigated, and the origin of the long-standing discrepancy between experimental measurements and previous theoretical predictions for the lifetime of these states is elucidated. Although surface-plasmon excitation had been expected to reduce the image-state lifetime, we demonstrate that the subtle combination of the spatial variation of s-d polarization in Ag and the characteristic nonlocality of many-electron interactions near the surface yields surprisingly long image-state lifetimes, in agreement with experiment.