A vision-language pretrained transformer for versatile clinical respiratory disease applications.
Authors
Affiliations (12)
Affiliations (12)
- School of Software, Tsinghua University, Beijing, China.
- Institute for Brain and Cognitive Sciences, BNRist, Tsinghua University, Beijing, China.
- Department of Thoracic Surgery, China State Key Laboratory of Respiratory Disease and National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China. [email protected].
- Department of Thoracic Surgery, China State Key Laboratory of Respiratory Disease and National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
- Department of Radiology, Guizhou Provincial People's Hospital, Guiyang, China.
- Department of Radiology, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China.
- Department of Radiology, Beijing Tiantan Hospital, Capital Medical University, Beijing, China.
- Tiantan Image Research Center, China National Clinical Research Center for Neurological Diseases, Beijing, China.
- Institute for Brain and Cognitive Sciences, BNRist, Tsinghua University, Beijing, China. [email protected].
- Department of Thoracic Surgery, China State Key Laboratory of Respiratory Disease and National Clinical Research Center for Respiratory Disease, The First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China. [email protected].
- School of Software, Tsinghua University, Beijing, China. [email protected].
- Institute for Brain and Cognitive Sciences, BNRist, Tsinghua University, Beijing, China. [email protected].
Abstract
General artificial intelligence models have unique challenges in clinical practice when applied to diverse modalities and complex clinical tasks. Here we present MedMPT, a versatile, clinically oriented pretrained model tailored for respiratory healthcare, trained on 154,274 pairs of chest computed-tomography scans and radiograph reports. MedMPT adopts self-supervised learning to acquire medical insights and is capable of handling multimodal clinical data and supporting various clinical tasks aligned with clinical workflows. We evaluate the performance of MedMPT on a broad spectrum of chest-related pathological conditions, involving common medical modalities such as computed tomography images, radiology reports, laboratory tests and drug relationship graphs. MedMPT consistently outperforms the state-of-the-art multimodal pretrained models in the medical domain, achieving significant improvements in diverse clinical tasks. Extensive analysis indicates that MedMPT effectively harnesses the potential of medical data, showing both data and parameter efficiency and offering explainable insights for decision-making. MedMPT highlights the potential of multimodal pretrained models in the realm of general-purpose artificial intelligence for clinical practice.