Collagen is the primary protein component in mammalian connective tissues. Over the last 20 years, evidence has mounted that collagen matrices exhibit substantial heterogeneity in their hierarchical structures and that this heterogeneity plays important roles in both structure and function. Herein, an overview of studies addressing the nanoscale compositional and structural heterogeneity is provided and connected to work exploring the mechanical implications for a number of tissues.