Quality of life (QOL) has become an integral part of the modern assessment of cancer treatment in Western society. However, little is known about the QOL of Chinese breast cancer survivors. To evaluate the long-term impact of medical and socio-demographic factors on survivors' QOL, we conducted a population-based study of 1065 breast cancer survivors in Shanghai, China. The mean age at diagnosis was 48.1 years and the median survival time was 4.3 years for the study participants. The Generic Quality of Life Inventory was used to assess survivors' QOL. Multiple linear regression models were employed to analyze the associations of QOL outcomes with socio-demographic and medical factors. The results revealed that recurrence status, time since diagnosis, marital status, income and education all had an independent, significant association with overall QOL and differential domains of QOL. Age at diagnosis exhibited a dual effect on QOL, positively associated with material well-being and negatively associated with physical well-being. Stage of disease was only associated with social well-being, while the type of surgery was related to material well-being alone. No associations between QOL and chemotherapy or radiotherapy were found. Further, analyses by survival intervals suggested a domain-specific order of recovery of QOL after cancer treatment. These results fill gaps in the limited literature, and provide valuable information for physicians to target the specific needs of Chinese women with breast cancer, choose appropriate interventions at the optimal time, and develop strategies accordingly in terms of improvement of patient's QOL.