A multi-modal foundation model for brain disease diagnosis and medical imaging.
Authors
Affiliations (10)
Affiliations (10)
- Department of Automation, BNRist, Tsinghua University, Beijing 100084, China.
- Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China.
- School of Information Science and Technology, Fudan University, Shanghai 200438, China.
- Department of Radiology, Chinese PLA General Hospital, Beijing 100039, China.
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen 518071, China.
- Department of Radiology, Beijing Tiantan Hospital, Capital Medical University, Beijing 100070, China.
- Tiantan Image Research Center, China National Clinical Research Center for Neurological Diseases, Beijing 100070, China.
- Beijing Friendship Hospital, Capital Medical University, Beijing 100050, China.
- Cancer Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Beijing 100730, China.
- School of Software, BNRist, Tsinghua University, Beijing 100084, China.
Abstract
The precise and comprehensive diagnosis of complex brain disorders relies on non-invasive computed tomography (CT) and magnetic resonance imaging (MRI) in conjunction with multi-modal clinical information. Here, we present Brainfound, a multi-modal foundation model for brain medical imaging that integrates image-text contrastive learning with a diffusion-based generative framework. The model was pre-trained on more than 3 million brain CT slices and 7 million brain MRI slices paired with clinical reports. In multi-center evaluations, Brainfound demonstrates state-of-the-art performance across seven tasks, including brain disease diagnosis, lesion segmentation, MRI enhancement, cross-modality translation, automatic report generation, zero-shot disease classification, and human-AI dialogue. It substantially outperforms leading models in automated report generation and clinical question answering for brain imaging, and its performance approaches that of expert physicians. These findings highlight the potential of Brainfound for accelerating diagnosis, support treatment decisions, and advance human-in-the-loop brain health care.