The traditional lipid profile differs by sex hormone levels. However, associations of sex hormones with lipoprotein subfractions, which may more accurately represent metabolic pathways to atherosclerosis, are not well studied. We quantified the cross-sectional associations of endogenous sex hormones with lipoprotein subfractions in 3143 men and 2038 postmenopausal women who were not on hormone replacement therapy, aged 45 to 84 years, in the Multi-Ethnic Study of Atherosclerosis baseline examination. Particle sizes and numbers of very low-density (VLDL), low-density (LDL), and high-density (HDL) lipoproteins were measured by nuclear magnetic resonance. In both men and women, after multivariable adjustment, higher sex hormone binding globulin (SHBG) levels are associated with smaller, fewer VLDL; larger, fewer LDL; and larger, more numerous HDL particles, whereas higher endogenous estradiol levels are associated with smaller VLDL and smaller, more numerous HDL and LDL particles (all P < .05). Testosterone (adjusted for SHBG) is associated with smaller VLDL particles in men but not women (sex difference P = .040). Higher dehydroepiandrosterone levels are associated with more numerous, smaller VLDL particles only in women (sex difference P = .030, .004, respectively). In conclusion, we found sex differences in the association of endogenous androgens with lipoprotein particle sizes and numbers. Higher endogenous estradiol, but lower SHBG, is associated with a more atherogenic lipoprotein particle profile. These findings highlight the potential to improve the lipoprotein profile with sex hormones, but emphasize the intricacies of the interactions.