Understanding the genetic architecture of human traits is of key biological, medical and evolutionary importance[1]. Despite much progress, little is known about how genetic architecture varies across the trait continuum and, in particular, if it differs in the tails of complex traits, where disease often occurs. Here, applying a novel approach based on polygenic scores, we reveal striking departures from polygenic architecture across 148 quantitative trait tails, consistent with distinct concentrations of high-impact rare alleles in one or both tails of most of the traits. We demonstrate replication of these results across ancestries, cohorts, repeat measures, and using an orthogonal family-based approach[2]. Furthermore, trait tails with inferred enrichment of rare alleles are associated with more exome study hits, reduced fecundity, advanced paternal age, and lower predictive accuracy of polygenic scores. Finally, we find evidence of ongoing selection consistent with the observed departures in polygenicity and demonstrate, via simulation, that traits under stabilising selection are expected to have tails enriched for rare, large-effect alleles. Overall, our findings suggest that while common variants of small effect likely account for most of the heritability in complex traits[3], rare variants of large effect are often more important in the trait tails, particularly among individuals at highest risk of disease. Our study has implications for rare variant discovery, the utility of polygenic scores, the study of selection in humans, and for the relative importance of common and rare variants to complex traits and diseases.