Multi-view contrastive learning and symptom extraction insights for medical report generation.
Authors
Affiliations (8)
Affiliations (8)
- Department of Orthopedics, The First Affiliated Hospital, Zhejiang University, #79 Qingchun Road, Hangzhou, Zhejiang Province, 310003, People's Republic of China.
- School of Mathematical Sciences, Zhejiang University, # 866 Yuhangtang Road, Hangzhou, Zhejiang Province, 310058, People's Republic of China.
- Department of Orthopedics, The Second Affiliated Hospital of Zhejiang Chinese Medical University, Xinhua Hospital of Zhejiang Province, Hangzhou, Zhejiang Province, 310003, People's Republic of China.
- Department of Orthopedics, Third Xiangya Hospital, Central South University, #138 Tongzi Po RoadHunan Province, Changsha, 410013, People's Republic of China.
- Zhejiang University School of Medicine, #866 Yuhangtang Road, Hangzhou, Zhejiang Province, 3100058, People's Republic of China.
- Department of Orthopedic Physical Therapy, Faculty of Physical Therapy, Nahda University in Beni Suef, Beni Suef, Egypt.
- School of Mathematical Sciences, Zhejiang University, # 866 Yuhangtang Road, Hangzhou, Zhejiang Province, 310058, People's Republic of China. [email protected].
- Department of Orthopedics, The First Affiliated Hospital, Zhejiang University, #79 Qingchun Road, Hangzhou, Zhejiang Province, 310003, People's Republic of China. [email protected].
Abstract
The task of generating medical reports automatically is of paramount importance in modern healthcare, offering a substantial reduction in the workload of radiologists and accelerating the processes of clinical diagnosis and treatment. Current challenges include handling limited sample sizes and interpreting intricate multi-modal and multi-view medical data. In order to improve the accuracy and efficiency for radiologists, we conducted this investigation. This study aims to present a novel methodology for medical report generation that leverages Multi-View Contrastive Learning (MVCL) applied to MRI data, combined with a Symptom Consultant (SC) for extracting medical insights, to improve the quality and efficiency of automated medical report generation. We introduce an advanced MVCL framework that maximizes the potential of multi-view MRI data to enhance visual feature extraction. Alongside, the SC component is employed to distill critical medical insights from symptom descriptions. These components are integrated within a transformer decoder architecture, which is then applied to the Deep Wrist dataset for model training and evaluation. Our experimental analysis on the Deep Wrist dataset reveals that our proposed integration of MVCL and SC significantly outperforms the baseline model in terms of accuracy and relevance of the generated medical reports. The results indicate that our approach is particularly effective in capturing and utilizing the complex information inherent in multi-modal and multi-view medical datasets. The combination of MVCL and SC constitutes a powerful approach to medical report generation, addressing the existing challenges in the field. The demonstrated superiority of our model over traditional methods holds promise for substantial improvements in clinical diagnosis and automated report generation, indicating a significant stride forward in medical technology.