Schizophrenia and bipolar disorder (BD) may be disorders of accelerated aging. Direct comparison of healthy aging populations with schizophrenia and BD patients across the adult lifespan may help inform this theory. In total, 225 individuals (91 healthy controls, 81 schizophrenia, 53 euthymic BD) underwent 3T T1-weighted magnetic resonance imaging, diffusion tensor imaging, and cognitive testing. We analyzed associations among age, diagnosis, and cognition with cortical thickness and fractional anisotropy (FA) using general linear models. We then assessed "brain age" using a random forest algorithm, which was also assessed in an independent sample (n = 147). Participants with schizophrenia had lower cortical thickness and FA compared with the other two groups, most prominently in fronto-temporal circuitry. These brain changes were more evident in younger participants than in older ones, yet were associated with cognitive performance independent of diagnosis. Predicted age was 8 years greater than chronological age in individuals with schizophrenia in the first sample and 6 years greater in the second sample. Predicted and chronological age were not different in BD. Differences in brain circuitry are present from illness onset most prominently in schizophrenia and to a lesser extent in BD. These results support a non-progressive "early hit" hypothesis/etiology of illness in the major psychoses. Brain age differences support the hypothesized early aging mechanism in schizophrenia but not in BD.