Our aim was to examine associations between leisure time physical activity and risk of non-vertebral fractures in men and women aged ≥55 years, with focus on the anatomical fracture location. Self-reported physical activity was registered in 3,450 men and 4,072 women aged 55-97 years at baseline in the Tromsø Study, Norway, in 1994-1995. Non-vertebral fractures were registered through December 31, 2009. During 75,546 person-years at risk, 1,693 non-vertebral fractures were identified. Risk of any non-vertebral fracture decreased with increasing physical activity level in men (P (trend) = 0.006) and non-significantly in women (P (trend) = 0.15), after adjustment for age, body mass index, height, smoking, and previous fracture. The reduced fracture risk was due to a reduced risk in the weight-bearing skeleton, particular at the hip, whereas risk of fracture in the non-weight-bearing skeleton was not related to physical activity. At weight-bearing sites, an inverse relationship between physical activity and fracture risk was present in both sexes (P (trend) ≤ 0.013). Compared with sedentary subjects, the most active men and women had respectively 37% (HR = 0.63, 95% CI: 0.45, 0.88) and 23% (HR = 0.77, 95% CI: 0.62, 0.95) reduced fracture risk in the weight-bearing skeleton. Physical activity is associated with reduced fracture risk at weight-bearing sites, with no associations at non-weight-bearing sites, in both sexes. Habitual physical activity is an important amendable approach to prevent hip fracture.