Post-deployment Monitoring of AI Performance in Intracranial Hemorrhage Detection by ChatGPT.
Authors
Affiliations (4)
Affiliations (4)
- Department of Radiology, Baylor College of Medicine, Houston, Texas (E.R.).
- Department Of Radiology, Division of Vascular and Interventional Radiology, Baylor College of Medicine, Houston, Texas (M.A., M.G.R.).
- Radiology Partners Research Institute, El Segundo, California (S.C., N.K., S.K., J.P., N.R., W.W., J.Y., C.Z., P.R.).
- Department Of Radiology, Division of Vascular and Interventional Radiology, Baylor College of Medicine, Houston, Texas (M.A., M.G.R.). Electronic address: [email protected].
Abstract
To evaluate the post-deployment performance of an artificial intelligence (AI) system (Aidoc) for intracranial hemorrhage (ICH) detection and assess the utility of ChatGPT-4 Turbo for automated AI monitoring. This retrospective study evaluated 332,809 head CT examinations from 37 radiology practices across the United States (December 2023-May 2024). Of these, 13,569 cases were flagged as positive for ICH by the Aidoc AI system. A HIPAA (Health Insurance Portability and Accountability Act) -compliant version of ChatGPT-4 Turbo was used to extract data from radiology reports. Ground truth was established through radiologists' review of 200 randomly selected cases. Performance metrics were calculated for ChatGPT, Aidoc and radiologists. ChatGPT-4 Turbo demonstrated high diagnostic accuracy in identifying intracranial hemorrhage (ICH) from radiology reports, with a positive predictive value of 1 and a negative predictive value of 0.988 (AUC:0.996). Aidoc's false positive classifications were influenced by scanner manufacturer, midline shift, mass effect, artifacts, and neurologic symptoms. Multivariate analysis identified Philips scanners (OR: 6.97, p=0.003) and artifacts (OR: 3.79, p=0.029) as significant contributors to false positives, while midline shift (OR: 0.08, p=0.021) and mass effect (OR: 0.18, p=0.021) were associated with a reduced false positive rate. Aidoc-assisted radiologists achieved a sensitivity of 0.936 and a specificity of 1. This study underscores the importance of continuous performance monitoring for AI systems in clinical practice. The integration of LLMs offers a scalable solution for evaluating AI performance, ensuring reliable deployment and enhancing diagnostic workflows.