The adsorption of rare gases on metal surfaces serves as the paradigm of weak adsorption where it is typically assumed that the adsorbate occupies maximally coordinated hollow sites. Density-functional theory calculations using the full-potential linearized augmented plane wave method for Xe adatoms on Mg(0001), Al(111), Ti(0001), Cu(111), Pd(111), and Pt(111), show, however, that Xe prefers low coordination on-top sites in all cases. We identify the importance of polarization and a site-dependent Pauli repulsion in actuating the site preference and the principle nature of the rare-gas atom-metal surface interaction.