Limited research has been done on the relationships between childhood factors and adult physical health related quality of life, with the underlying pathways not fully elucidated. Data from 2292 participants of the British 1946 birth cohort were used to examine the relationship of childhood characteristics and family environment with principal component summary (PCS) scores and the physical functioning (PF) subscale of the SF-36 at age 60-64 years. Impaired physical functioning was defined as the lowest quartile scores in the PF subscale. Childhood factors (father in manual social class versus non-manual (β = -2.34; 95%CI: -3.39, -1.28) and poor maternal health versus good/excellent maternal health (β = -6.18; -8.78, -3.57)) were associated with lower PCS scores at 60-64 years. Adult health behaviours (increasing BMI, lifelong smoking, and lower physical activity) at 53 years were identified as strong risk factors for lower PCS scores. After adjusting for these factors and education level (N = 1463), only poor maternal health remained unattenuated (β = -5.07; -7.62, -2.51). Similarly poor maternal health doubled the risk of reporting impaired PF (Odds ratio = 2.45; 95%CI: 1.39, 4.30); serious illness in childhood (OR = 1.44; 1.01, 2.06) and lower educational level attained were also risk factors for impaired PF (N = 1526). While findings suggest the influence of father's social class on physical health related quality of life are mediated by modifiable adult social factors and health behaviours; health professionals should also be mindful of the inter-generational risk posed by poor maternal health on the physical health related quality of life of her offspring almost five decades later.