Researchers found ChatGPT-4 Turbo could efficiently monitor the performance of Aidoc's ICH detection AI across real-world radiology practices.
Key Details
- 1Study used ChatGPT-4 Turbo to monitor Aidoc's intracranial hemorrhage (ICH) detection on 332,809 head CT exams from 37 sites.
- 2Compared LLM data extraction to a ground-truth set of 1,000 radiologist-labeled reports.
- 3LLM achieved 0.995 accuracy, 0.99 AUC, PPV of 1, and NPV of 0.98.
- 4Discordant cases were mostly due to Aidoc overcalls; only 0.5% due to LLM extraction error.
- 5Aidoc AI's performance varied across CT scanner models and was influenced by scanner manufacturer, exam artifacts, and patient symptoms.
- 6Authors emphasized that LLM monitoring is cost-effective and scalable compared to manual review.
Why It Matters
Reliable, scalable solutions for performance monitoring are crucial as AI tools become increasingly common in radiological practice. Using large language models like ChatGPT could streamline postdeployment quality assurance and ensure ongoing diagnostic accuracy.
Source
AuntMinnie
Related News
•AuntMinnie
Study: Computer Vision Models Best LLMs in Chest CT Breast Abnormality Detection
Computer vision models (CVMs) surpass large language models (LLMs) in accurately labeling incidental breast abnormalities on chest CT scans.
•Cardiovascular Business
Radiology Maintains Lead in FDA-Cleared AI Algorithms, Cardiology Follows
Radiology remains the top specialty for FDA-cleared AI, with cardiology as a strong second, particularly in cardiovascular imaging.
•AuntMinnie
Deep Learning Models Rival Radiologists for Pancreatic Cancer Detection on CT
Deep-learning models achieved comparable or superior accuracy to experienced radiologists in detecting pancreatic cancer on CT scans, especially for small tumors.