AI assisted detection of large vessel occlusion on non-contrast CT: multinational validation and reader study.
Authors
Affiliations (9)
Affiliations (9)
- Department of Radiology, Seoul National University Bundang Hospital, Seoul National University College of Medicine, Seongnam, Korea (the Republic of).
- Artificial Intelligence Research Center, JLK, Seoul, Korea (the Republic of).
- Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts, USA.
- Department of Radiology, Brigham and Women's Hospital, Boston, Massachusetts, USA.
- Department of Neurology and Cerebrovascular Center, Seoul National University Bundang Hospital, Seoul National University College of Medicine, Seongnam, Korea (the Republic of).
- Department of Neurology, Korea University Guro Hospital, Seoul, Korea (the Republic of).
- Department of Radiology, Seoul National University Hospital, Seoul National University College of Medicine, Seoul, Korea (the Republic of).
- Department of Neurology, Northwestern University Feinberg School of Medicine, Chicago, Illinois, USA.
- Department of Neurology and Cerebrovascular Center, Seoul National University Bundang Hospital, Seoul National University College of Medicine, Seongnam, Korea (the Republic of) [email protected].
Abstract
Early detection of large vessel occlusion (LVO) on non-contrast CT (NCCT) could accelerate stroke triage, but NCCT based artificial intelligence (AI) algorithms lack multinational validation and evidence of impact on clinician performance. We aimed to validate a machine learning based LVO detection algorithm across multinational cohorts and evaluate its impact on clinicians' diagnostic performance. In this retrospective study, an AI algorithm was validated using independent cohorts from Korea (n=723; consecutive; 127 with LVO) and the US (n=240; case-control; 120 with LVO). Standalone performance was assessed using the area under the receiver operating characteristic curve (AUC), sensitivity, and specificity. A multi-reader, multi-case crossover study involving eight physicians evaluated diagnostic performance with and without AI assistance. Clinical utility was quantified using net reclassification improvement (NRI), number needed to screen (NNS), and benefit-to-harm ratio (BHR). Standalone AI achieved AUC values of 0.963 (95% CI 0.946 to 0.975) in the Korean cohort and 0.899 (95% CI 0.858 to 0.939) in the US cohort. AI assistance significantly increased pooled AUC values from 0.718 to 0.852, sensitivity from 46.6% to 63.7%, and specificity from 91.9% to 94.9% (all P<0.001). AI gave an NRI of 5.5%, an NNS of 18.2, and a BHR of 2.89. Automation bias analysis showed a reliance level BHR of 9.25. This multinational validation confirmed the robustness and generalizability of a machine learning based LVO detection algorithm on NCCT. AI assistance significantly improved clinicians' sensitivity and diagnostic accuracy, suggesting its potential as a supportive tool in acute stroke triage.