Simple Summary: Sexual dysfunction (SD) after rectal cancer surgery is common, inadequately discussed, multifactorial and often untreated. It is well known that postoperative SD is caused by nerve damage or vascular injury of the vasa nervosa during surgery or perioperative therapy. The majority of the studies have focused on male dysfunction, missing female data or explored SD only as a secondary outcome. Moreover, the impact on quality of life is rarely described in previous literature. The aim of this study was to evaluate risk factors for worse sexual function and quality of life after multimodal therapy and surgery for locally advanced rectal cancer. Purpose: Sexual function is crucial for the quality of life and can be highly affected by preoperative therapy and surgery. The aim of this study was to identify potential risk factors for poor sexual function and quality of life. Methods: Female patients were asked to complete the Female Sexual Function Index (FSFI-6). Male patients were demanded to answer the International Index of Erectile Function (IIEF-5). Results: In total, 79 patients filled in the questionary, yielding a response rate of 41.57%. The proportion of women was represented by 32.91%, and the median age was 76.0 years (66.0–81.0). Sexual dysfunction appeared in 88.46% of female patients. Severe erectile dysfunction occurred in 52.83% of male patients. Univariate analysis showed female patients (OR: 0.17, 95%CI: 0.05–0.64, p = 0.01), older age (OR: 0.34, 95%CI 0.11–1.01, p = 0.05), tumor localization under 6cm from the anal verge (OR: 4.43, 95%CI: 1.44–13.67, p = 0.01) and extension of operation (APR and ISR) (OR: 0.13, 95%CI: 0.03–0.59, p = 0.01) as significant risk factors for poor outcome. Female patients (OR: 0.12, 95%CI: 0.03–0.62, p = 0.01) and tumors below 6 cm from the anal verge (OR: 4.64, 95%CI: 1.18–18.29, p = 0.03) were shown to be independent risk factors for sexual dysfunction after multimodal therapy in the multivariate analysis. Quality of life was only affected in the case of extensive surgery (p = 0.02). Conclusion: Higher Age, female sex, distal tumors and extensive surgery (APR, ISR) are revealed risk factors for SD in this study. Quality of life was only affected in the case of APR or ISR. [ABSTRACT FROM AUTHOR]