Transformer-based Fusion of Longitudinal Multimodal Radiomic Features from Chest Radiography and CT in COVID-19.
Authors
Affiliations (5)
Affiliations (5)
- Department of Radiology, Columbia University, Street Address, Alianza Dominicana Cultural Center, 5th Fl, New York, NY.
- Department of Radiology, University of Pennsylvania, Philadelphia, Pa.
- Department of Pathology and Laboratory Medicine, University of Pennsylvania, Philadelphia, Pa.
- Department of Medicine, University of Pennsylvania, Philadelphia, Pa.
- Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania, Philadelphia, Pa.
Abstract
Purpose To evaluate the feasibility of a transformer structure for fusing longitudinal multimodal radiomic features from chest radiographs (CXRs) and CT images to predict outcomes and identify associated clinical events in patients with COVID-19. Materials and Methods This retrospective study analyzed de-identified longitudinal CXRs and CT images from polymerase chain reaction-confirmed COVID-19 patients. Proprietary patient data (Site 1) were collected between July 2020 and May 2021, and open-access patient data (obtained before February 1, 2020) were collected from Site 2. Clinical outcomes included mortality, intensive care unit (ICU) admission, and ventilator use during any followup visit. Radiomic features were extracted from lung regions in CXRs and CT images using the Cancer Imaging Phenomics Toolkit and integrated using a transformer-based model. Patient data were partitioned into training, validation, and test sets (65:15:20). Subgroup analyses were performed across sex, site, and modality. Model performance was assessed using area under the receiver operating characteristic curve (AUC) and weighted AUC scores, with statistical significance assessed using Student's <i>t</i> tests. Results The study included 2274 patients (946 from Site 1, 1328 from Site 2; mean age, 59.84 ± 16.84 years, 1171 males). Weighted testing AUCs for predicting outcomes were 0.86 (95% CI: 0.85, 0.86) for mortality, 0.82 (95% CI: 0.81, 0.82) for ICU admission, and 0.86 (95% CI: 0.86, 0.87) for ventilator usage, outperforming models trained solely on cross-sectional data or single-modal data (<i>P</i> < .05). Conclusion Transformer-based fusion of longitudinal multimodal radiomic data effectively predicted clinical outcomes and events associated with COVID-19. © RSNA, 2026.