Multimodal Deep Learning with Routine Clinical Data for Recurrence Risk Stratification in HR<sup>+</sup>/HER2<sup>-</sup> Early Breast Cancer.
Authors
Affiliations (5)
Affiliations (5)
- Department of Pathology, West China Hospital, Sichuan University, Chengdu, China.
- Institute of Clinical Pathology, West China Hospital, Sichuan University, Chengdu, China.
- College of Computer Science, Sichuan University, Chengdu, China.
- Department of Medical Ultrasound, West China Hospital, Sichuan University, Chengdu, China.
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, China.
Abstract
Hormone receptor-positive/human epidermal growth factor receptor 2-negative (HR<sup>+</sup>/HER2<sup>-</sup>) early breast cancer (EBC) patients face long-term recurrence risk despite standard treatment. Current prognostic tools relying on clinicopathological factors or multigene assays have limited accuracy or accessibility. In this study, we developed a multimodal recurrence risk prediction (MRRP) model integrating routinely available clinical data, including whole-slide images (WISs), ultrasound (US) imaging and diagnostic reports, and structured clinical parameters. The MRRP model employs a hierarchical transformer-based fusion framework with innovative intra- and intermodality cross-attention mechanisms to dynamically integrate diverse feature representations. Using a well-curated cohort of 768 HR<sup>+</sup>/HER2<sup>-</sup> EBC patients with long-term follow-up, MRRP demonstrated superior prognostic performance (C-index = 0.840) compared to single-modality models, with robust time-dependent AUCs exceeding 0.85 at 3, 5, and 7 years. Ablation studies highlighted the central role of pathology features and the complementary value of US and clinical data. We further validated the optimal query selection strategies and evaluated different pretrained encoders, revealing complex modality interactions. To address real-world challenges of missing modality data, a learnable compensation mechanism was implemented, improving model robustness. Our study provides a clinically practical, AI-driven tool for precise risk stratification in HR<sup>+</sup>/HER2<sup>-</sup> EBC patients, facilitating individualized treatment and surveillance decisions without reliance on costly multi-omics data.