MuTriM: A multiscale deep learning model integrating longitudinal radiomics and pathomic features for predicting recurrence and adjuvant radiation benefit in breast cancer.
Authors
Affiliations (10)
Affiliations (10)
- Jiangsu Key Laboratory of Intelligent Medical Image Computing, Nanjing University of Information Science and Technology, Nanjing, China.
- Reproductive Center, Women's Hospital of Nanjing Medical University (Nanjing Women and Children's Healthcare Hospital), Nanjing, China.
- Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA.
- Department of Radiology, Guangdong Provincial People's Hospital (Guangdong Academy of Medical Sciences), Guangzhou, China.
- Department of Hematology and Medical Oncology, Emory University, Atlanta, GA, USA.
- Department of Pathology and Laboratory Medicine, Emory University, Atlanta, GA, USA.
- Key Laboratory of Breast Cancer in Shanghai, Department of Breast Surgery, Fudan University Shanghai Cancer Center, Shanghai, China.
- Department of Pathology, Fudan University Shanghai Cancer Center, Shanghai, China.
- Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA; Atlanta Veterans Administration Medical Center, Atlanta, GA, USA. Electronic address: [email protected].
- Wallace H. Coulter Department of Biomedical Engineering, Georgia Institute of Technology and Emory University, Atlanta, GA, USA. Electronic address: [email protected].
Abstract
We present MuTriM, a multimodal deep learning model integrating DCE-MRI and whole-slide pathology to predict survival and radiation benefit in breast cancer. MuTriM employs an attention-based cross-modal and cross-temporal fusion framework that concurrently integrates DCE-MRI radiomic features with pathomic features of cellular morphology on WSI. MuTriM was trained on the Fudan University Shanghai Cancer Center (FUSCC) cohort (N = 335) and externally tested on the TCGA cohort (N = 126) using matched DCE-MRI and WSI. MuTriM was prognostic of recurrence-free survival (RFS) in a subtype-agnostic setting (HR = 5.26, 95% CI, 1.69-16.4, p = 0.004; C-index = 0.75), outperforming unimodal models based on DCE-MRI alone (C-index = 0.65) or WSI alone (C-index = 0.70). MuTriM was also prognostic among HER2 + (HR = 6.82, 95% CI, 1.08-43.2, p = 0.04) and ER+ populations (HR = 4.61, 95% CI, 1.25-17.1, p = 0.02). A high MuTriM score was associated with improved RFS (HR = 0.15, 95% CI 0.03-0.87; P = 0.03), whereas a low MuTriM score showed no significant benefit (HR = 4.06, 95% CI 0.05-31; P = 0.53) (interaction P = 0.04). Transcriptomic analyses revealed that high-risk tumors showed suppressed cytotoxic T-cell activity and enrichment of extracellular matrix remodeling, neutrophil degranulation, and neuronal signaling, consistent with immune evasion. Upregulation of NRXN1, GABRA3, KCNK3, and KLK12 was linked to worse survival, whereas SYNGR3 and CD207 were associated with favorable outcomes. By unifying radiology, pathology, and transcriptomics, MuTriM provides an interpretable multimodal framework for breast cancer prognosis and treatment guidance.