Abstract Objective To develop and validate a deep learning model based on multi-lesion and time series CT images in predicting overall survival (OS) in patients with stage IV gastric cancer (GC) receiving anti-HER2 targeted therapy. Methods A total of 207 patients were enrolled in this multicenter study, with 137 patients for retrospective training and internal validation, 33 patients for prospective validation, and 37 patients for external validation. All patients received anti-HER2 targeted therapy and underwent pre- and post-treatment CT scans (baseline and at least one follow-up). The proposed deep learning model evaluated the multiple lesions in time series CT images to predict risk probabilities. We further evaluated and validated the risk score of the nomogram combining a two-follow-up lesion-based deep learning model (LDLM-2F), tumor markers, and clinical information for predicting the benefits from treatment (Nomo-LDLM-2F). Results In the internal validation and prospective cohorts, the one-year AUCs for Nomo-LDLM-2F using the time series medical images and tumor markers were 0.894 (0.728–1.000) and 0.809 (0.561–1.000), respectively. In the external validation cohort, the one-year AUC of Nomo-LDLM-2F without tumor markers was 0.771 (0.510–1.000). Patients with a low Nomo-LDLM-2F score derived survival benefits from anti-HER2 targeted therapy significantly compared to those with a high Nomo-LDLM-2F score (all p