SARS-CoV-2 variants acquire mutations in spike that promote immune evasion and impact other properties that contribute to viral fitness such as ACE2 receptor binding and cell entry. Knowledge of how mutations affect these spike phenotypes can provide insight into the current and potential future evolution of the virus. Here we use pseudovirus deep mutational scanning to measure how >9,000 mutations across the full XBB.1.5 and BA.2 spikes affect ACE2 binding, cell entry, or escape from human sera. We find that mutations outside the receptor-binding domain (RBD) have meaningfully impacted ACE2 binding during SARS-CoV-2 evolution. We also measure how mutations to the XBB.1.5 spike affect neutralization by serum from individuals who recently had SARS-CoV-2 infections. The strongest serum escape mutations are in the RBD at sites 357, 420, 440, 456, and 473-however, the antigenic impacts of these mutations vary across individuals. We also identify strong escape mutations outside the RBD; however many of them decrease ACE2 binding, suggesting they act by modulating RBD conformation. Notably, the growth rates of human SARS-CoV-2 clades can be explained in substantial part by the measured effects of mutations on spike phenotypes, suggesting our data could enable better prediction of viral evolution.