The antagonistic interaction with host restriction proteins is a major driver of evolutionary change for viruses. We previously reported that polymorphisms of the TRIM5α B30.2/SPRY domain impacted the level of SIVsmm viremia in rhesus macaques. Viremia in macaques homozygous for the non-restrictive TRIM5α allele TRIM5(Q) was significantly higher than in macaques expressing two restrictive TRIM5alpha alleles TRIM5(TFP/TFP) or TRIM5(Cyp/TFP). Using this model, we observed that despite an early impact on viremia, SIVsmm overcame TRIM5α restriction at later stages of infection and that increasing viremia was associated with specific amino acid substitutions in capsid. Two amino acid substitutions (P37S and R98S) in the capsid region were associated with escape from TRIM5(TFP) restriction and substitutions in the CypA binding-loop (GPLPA87-91) in capsid were associated with escape from TRIM5(Cyp). Introduction of these mutations into the original SIVsmE543 clone not only resulted in escape from TRIM5α restriction in vitro but the P37S and R98S substitutions improved virus fitness in macaques with homozygous restrictive TRIM(TFP) alleles in vivo. Similar substitutions were observed in other SIVsmm strains following transmission and passage in macaques, collectively providing direct evidence that TRIM5α exerts selective pressure on the cross-species transmission of SIV in primates.