High Accuracy but Low Explainability: The Challenge of XAI in Multiple Sclerosis Assessment from MRI Radiology Reports.
Authors
Affiliations (5)
Affiliations (5)
- MRI unit, Radiology department. HT medica, Jaén, SPAIN. Electronic address: [email protected].
- NLP department. HT medica. Jaén, SPAIN. Electronic address: [email protected].
- Diagnostic and Interventional Neuroradiology Unit. HT medica, Ávila. SPAIN. Electronic address: [email protected].
- Division of Neuroradiology, Hospital of the University of Pennsylvania, Philadelphia, PA, USA. Electronic address: [email protected].
- MRI unit, Radiology department. HT medica, Jaén, SPAIN. Electronic address: [email protected].
Abstract
Timely identification of disease progression and/or active lesions in multiple sclerosis (MS) is essential for clinical management. Radiology reports often contain complex language, making consistent interpretation challenging. We developed a natural language processing (NLP)-based tool to assist radiologists in detecting MS-related changes and evaluated its explainability. To assess the performance and interpretability of NLP algorithms in identifying disease progression and/or active lesions in MRI reports of MS patients. A retrospective study included 600 MRI reports labeled for MS progression and/or active lesions (January 2013-July 2022). Five hundred reports were used to fine-tune RoBERTa-based models; 100 served as the test set. A prospective evaluation was conducted on 122 reports. Explainability was assessed using the LIME tool and radiologist feedback. Retrospective accuracy was 87% for new/enlarged lesions and 96% for active lesions. Prospective accuracy improved to 94.26% and 99.18%, respectively. LIME-based interpretability yielded radiologist agreement rates of 53.2% (new/enlarged lesions) and 52.5% (active lesions). Our NLP tools demonstrated high accuracy in detecting MS-related MRI findings. However, explainability remains limited, underscoring the need for more intuitive interpretability methods to support clinical integration.