Group-living species show a diversity of social organization, from simple mated pairs to complex communities of interdependent individuals performing specialized tasks. The advantages of living in cooperative groups are well understood, but why some species breed in small aggregations while others evolve large, complex groups with clearly divided roles is unclear. We address this problem by reconstructing the evolutionary pathways to cooperative breeding across 4,730 bird species. We show that differences in the way groups form at the origin of cooperative breeding predicts the level of group complexity that emerges. Groups that originate through the retention of offspring have a clear reproductive divide with distinct breeder and helper roles. This is associated with reproductive specialization, where breeders invest more in fecundity and less in care. In contrast, groups formed through the aggregation of unrelated adults are smaller and lack specialization. These results help explain why some species have not transitioned beyond simple groups while others have taken the pathway to increased group complexity.