Supported by histological and genetic evidence implicating myelin, neuroinflammation and oligodendrocyte dysfunction in schizophrenia spectrum disorders (SZ), diffusion tensor imaging (DTI) studies have consistently shown white matter (WM) abnormalities when compared to healthy controls (HC). The diagnostic specificity remains unclear, with bipolar disorders (BD) frequently conceptualized as a less severe clinical manifestation along a psychotic spectrum. Further, the age-related dynamics and possible sex differences of WM abnormalities in SZ and BD are currently understudied. Using tract-based spatial statistics (TBSS) we compared DTI-based microstructural indices between SZ (n = 128), BD (n = 61), and HC (n = 293). We tested for age-by-group and sex-by-group interactions, computed effect sizes within different age-bins and within genders. TBSS revealed global reductions in fractional anisotropy (FA) and increases in radial (RD) diffusivity in SZ compared to HC, with strongest effects in the body and splenium of the corpus callosum, and lower FA in SZ compared to BD in right inferior longitudinal fasciculus and right inferior fronto-occipital fasciculus, and no significant differences between BD and HC. The results were not strongly dependent on age or sex. Despite lack of significant group-by-age interactions, a sliding-window approach supported widespread WM involvement in SZ with most profound differences in FA from the late 20 s.