Insulin resistance is associated with metabolic inflexibility, impaired switching of substrate oxidation from fatty acids to glucose in response to insulin. Impaired switching to fat oxidation in response to a high-fat diet (HFD) is hypothesized to contribute to insulin resistance. The objective of this study was to test the hypothesis that defects in substrate switching in response to insulin and a HFD are linked to reduced mitochondrial biogenesis and occur before the development of diabetes. Metabolic flexibility was measured in young sedentary men with (n = 16) or without (n = 34) a family history of diabetes by euglycemic-hyperinsulinemic clamp. Flexibility correlated with fat oxidation measured in a respiratory chamber after a 3-day HFD. Muscle mitochondrial content was higher in flexible subjects with high fat oxidation after a HFD and contributed 49% of the variance. Subjects with a family history of diabetes were inflexible and had reduced HFD-induced fat oxidation and muscle mitochondrial content but did not differ in the amount of body or visceral fat. Metabolic inflexibility, lower adaptation to a HFD, and reduced muscle mitochondrial mass cluster together in subjects with a family history of diabetes, supporting the role of an intrinsic metabolic defect of skeletal muscle in the pathogenesis of insulin resistance.