Only few studies have investigated the impact of surgery for rectal cancer on sexual function. Little of that research included quality of life (QoL) aspects and hardly any study analyzed the impact of age, gender and type of surgery on sexual function. The aim of the presented study was to address these issues. Over a 5 y period, EORTC-QLQ-C-30 and a tumor-specific module were prospectively administered to patients before surgery, at discharge, 3, 6, 12 and 24 months postoperatively. Comparisons were made between patients receiving abdominoperineal resection (APR), anterior resection (AR) with or without Pouch and Sigmoid resection. Furthermore, effects of surgery on female and male patients, and age groups were analyzed. A total of 819 patients participated in the study: 412 were males and 407 were females. The groups were comparable in terms of adjuvant treatment, tumor stage and histology. Patients after APR and AR with Pouch had worst sexual function. Men reported significantly more difficulties with sexual enjoyment; furthermore, over time, sexual problems created high levels of strain in men that were worse than baseline levels in the early postoperative period. These problems tended to remain. Patients aged 69 y and younger scored higher for problems with loss of sexual function and sexuality-related strain than patients aged 70 y and older. The findings in this study confirm that QoL changes postsurgery and that factors like type of surgery, gender and age have tremendous impact on sexual function and sexual enjoyment. APR and AR with Pouch affect sexual function more than AR and resection of the lower sigmoid. Through impaired sexual enjoyment, men are put more under strain than women. Patients aged 69 y and younger experience more stress through deteriorated sexual function.