Noroviruses are important human pathogens known to cause epidemic outbreaks of severe gastroenteritis in communities, military barracks, cruise ships, hospitals, and assisted living communities, resulting in over 267,000,000 annual infections worldwide. Diversity within the norovirus genus allows this virus to persist in human populations, although a single genocluster, the GII.4 noroviruses, currently accounts for approximately 80% of all infections. Noroviruses bind to the polymorphic histoblood group antigens (HBGAs), which act as the putative cellular receptor, and strains from different genoclusters bind various HBGAs. Human challenge studies using viruses from different genoclusters have demonstrated that norovirus immunity is complicated and probably confounded by pre-existing exposure histories and variable immune responses. Evidence for both short-term and long-term immunity has been demonstrated, but the molecular mechanisms mediating differential immune responses in the face of infection remain unclear. Studies with virus-like particles from the GII.4 genocluster demonstrated that variation in and around the receptor-binding domain results in differential HBGA binding and altered antigenicity. These observations suggest that the norovirus capsid evolves to evade the memory immune response while retaining its ability to bind any of several HBGAs. In this review, we discuss how evolution within the capsid drives receptor switching and allows escape from herd immunity.