Multimodal integration of longitudinal noninvasive diagnostics for survival prediction in immunotherapy using deep learning.
Authors
Affiliations (9)
Affiliations (9)
- Department of Machine Learning and Neural Computing, Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen 6525 GD, The Netherlands.
- Department of Radiology, The Netherlands Cancer Institute, Amsterdam 1066 CX, The Netherlands.
- GROW School for Oncology and Developmental Biology, Maastricht University, Maastricht 6229 ER, The Netherlands.
- Department of Laboratory Medicine, The Netherlands Cancer Institute, Amsterdam 1066 CX, The Netherlands.
- Department of Medical Oncology, The Netherlands Cancer Institute, Amsterdam 1066 CX, The Netherlands.
- Division of Molecular Oncology and Immunology, Oncode Institute, Amsterdam 1066 CX, The Netherlands.
- Department of Medical Oncology, Leiden University Medical Center, Leiden 2333 ZG, The Netherlands.
- Melanoma Clinic, Centre Hospitalier Universitaire Vaudois, Lausanne 1005, Switzerland.
- Faculty of Health Science, University of Southern Denmark, Odense 5230, Denmark.
Abstract
Immunotherapies have revolutionized the landscape of cancer treatments. However, our understanding of response patterns in advanced cancers treated with immunotherapy remains limited. By leveraging routinely collected noninvasive longitudinal and multimodal data with artificial intelligence, we could unlock the potential to transform immunotherapy for cancer patients, paving the way for personalized treatment approaches. In this study, we developed a novel artificial neural network architecture, multimodal transformer-based simple temporal attention (MMTSimTA) network, building upon a combination of recent successful developments. We integrated pre- and on-treatment blood measurements, prescribed medications, and CT-based volumes of organs from a large pan-cancer cohort of 694 patients treated with immunotherapy to predict mortality at 3, 6, 9, and 12 months. Different variants of our extended MMTSimTA network were implemented and compared to baseline methods, incorporating intermediate and late fusion-based integration methods. The strongest prognostic performance was demonstrated using a variant of the MMTSimTA model with area under the curves of 0.84 ± 0.04, 0.83 ± 0.02, 0.82 ± 0.02, 0.81 ± 0.03 for 3-, 6-, 9-, and 12-month survival prediction, respectively. Our findings show that integrating noninvasive longitudinal data using our novel architecture yields an improved multimodal prognostic performance, especially in short-term survival prediction. Our study demonstrates that multimodal longitudinal integration of noninvasive data using deep learning may offer a promising approach for personalized prognostication in immunotherapy-treated cancer patients.