Using scanning tunneling microscopy, we studied the diffusion of single Ag atoms within the Si(111)-(7 x 7) unit cell. A striking difference was observed in the time-dependent tunneling current spectra for Ag atoms moving in the unfaulted and faulted half unit cells, with a dual-time characteristic in the former but a single time in the latter. Our observations demonstrate the importance of the stacking fault in affecting the interaction between Ag atoms and the Si(111)-(7 x 7) surface and can be understood in terms of an asymmetric interplay between intrabasin and interbasin diffusion.