Development and Validation of an AI-Based Multimodal Model for Pathological Staging of Gastric Cancer Using CT and Endoscopic Images

Rationale and objectives: Accurate preoperative pathological staging of gastric cancer is crucial for optimal treatment selection and improved patient outcomes. Traditional imaging methods such as CT and endoscopy have limitations in staging accuracy.

Methods: This retrospective study included 691 gastric cancer patients treated from March 2017 to March 2024. Enhanced venous-phase CT and endoscopic images, along with postoperative pathological results, were collected. We developed three modeling approaches: (1) nine deep learning models applied to CT images (DeepCT), (2) 11 machine learning algorithms using handcrafted radiomic features from CT images (HandcraftedCT), and (3) ResNet-50-extracted deep features from endoscopic images followed by 11 machine learning algorithms (DeepEndo). The two top-performing models from each approach were combined into the Integrated Multi-Modal Model using a stacking ensemble method. Performance was assessed using ROC-AUC, sensitivity, and specificity.

Results: The Integrated Multi-Modal Model achieved an ROC-AUC of 0.933 (95% CI, 0.887-0.979) on the test set, outperforming individual models. Sensitivity and specificity were 0.869 and 0.840, respectively. Various evaluation metrics demonstrated that the final fusion model effectively integrated the strengths of each sub-model, resulting in a balanced and robust performance with reduced false-positive and false-negative rates.

Conclusion: The Integrated Multi-Modal Model effectively integrates radiomic and deep learning features from CT and endoscopic images, demonstrating superior performance in preoperative pathological staging of gastric cancer. This multimodal approach enhances predictive accuracy and provides a reliable tool for clinicians to develop individualized treatment plans, thereby improving patient outcomes.

Data availability: The data presented in this study are available on request from the corresponding author. The data are not publicly available due to ethical reasons. All code used in this study is based on third-party libraries and all custom code developed for this study is available upon reasonable request from the corresponding author.