Today we lack knowledge if size at birth and gestational age interact regarding postnatal growth pattern in children born at 32 gestational weeks or later. This population-based cohort study comprised 41,669 children born in gestational weeks 32-40 in Uppsala County, Sweden, between 2000 and 2015. We applied a generalized least squares model including anthropometric measurements at 1.5, 3, 4 and 5 years. We calculated estimated mean height, weight and BMI for children born in week 32 + 0, 35 + 0 or 40 + 0 with birthweight 50th percentile (standardized appropriate for gestational age, sAGA) or 3rd percentile (standardized small for gestational age, sSGA). Compared with children born sAGA at gestational week 40 + 0, those born sAGA week 32 + 0 or 35 + 0 had comparable estimated mean height, weight and BMI after 3 years of age. Making the same comparison, those born sSGA week 32 + 0 or 35 + 0 were shorter and lighter with lower estimated mean BMI throughout the whole follow-up period. Our findings suggest that being born SGA and moderate preterm is associated with short stature and low BMI during the first five years of life. The association seemed stronger the shorter gestational age at birth.