Background: Tumor deposits (TDs) are associated with poor prognosis in rectal cancer (RC). This study aims to develop and validate a deep learning (DL) model incorporating T2-MR image and clinical factors for the preoperative prediction of TDs in RC patients. Methods and methods: A total of 327 RC patients with pathologically confirmed TDs status from January 2016 to December 2019 were retrospectively recruited, and the T2-MR images and clinical variables were collected. Patients were randomly split into a development dataset (n = 246) and an independent testing dataset (n = 81). A single-channel DL model, a multi-channel DL model, a hybrid DL model, and a clinical model were constructed. The performance of these predictive models was assessed by using receiver operating characteristics (ROC) analysis and decision curve analysis (DCA). Results: The areas under the curves (AUCs) of the clinical, single-DL, multi-DL, and hybrid-DL models were 0.734 (95% CI, 0.674–0.788), 0.710 (95% CI, 0.649–0.766), 0.767 (95% CI, 0.710–0.819), and 0.857 (95% CI, 0.807–0.898) in the development dataset. The AUC of the hybrid-DL model was significantly higher than the single-DL and multi-DL models (both p < 0.001) in the development dataset, and the single-DL model (p = 0.028) in the testing dataset. Decision curve analysis demonstrated the hybrid-DL model had higher net benefit than other models across the majority range of threshold probabilities. Conclusions: The proposed hybrid-DL model achieved good predictive efficacy and could be used to predict tumor deposits in rectal cancer. Critical relevance statement: The proposed hybrid-DL model achieved good predictive efficacy and could be used to predict tumor deposits in rectal cancer. Key points: • Preoperative non-invasive identification of TDs is of great clinical significance. • The combined hybrid-DL model achieved good predictive efficacy and could be used to predict tumor deposits in rectal cancer. • A preoperative nomogram provides gastroenterologist with an accurate and effective tool. [ABSTRACT FROM AUTHOR]