Artificial Intelligence-assisted reader evaluation in acute CT head interpretation (AI-REACT): a multireader multicase study.

Authors

Abstract

BackgroundNon-contrast CT head scans (NCCTH) are the most frequently requested cross-sectional imaging in the Emergency Department. While AI tools have been developed to detect NCCTH abnormalities, most validation studies compare AI to radiologists, with limited evidence on the impact of AI assistance for other healthcare professionals. ObjectiveTo evaluate whether an AI-powered tool improves the accuracy, speed, and confidence of general radiologists, emergency clinicians, and radiographers in detecting critical abnormalities on NCCTH, and to assess the tools stand-alone performance and factors influencing diagnostic accuracy and efficiency. MethodsA retrospective dataset of 150 NCCTH (52 normal, 98 with critical abnormalities: intracranial haemorrhage, hypodensity, midline shift, mass effect, or skull fracture) was reviewed by 30 readers (10 radiologists, 15 emergency clinicians, 5 radiographers) from four NHS trusts. Each reader interpreted scans first unaided, then with the qER EU 2.0 AI tool, separated by a 2-week washout. Ground truth was established by consensus of two neuroradiologists. We assessed the stand-alone performance of qER and its effect on reader diagnostic accuracy, confidence, and interpretation speed. ResultsThe qER algorithm demonstrated strong diagnostic performance across most pathology subgroups (AUC 0.821-0.976). With AI assistance, pooled reader sensitivity for critically abnormal scans increased from 82.8% to 89.7% (+6.9%, 95% CI +1.4% to +10.6%, p<0.001), and for intracranial haemorrhage from 84.6% to 91.6% (+7.0%, 95% CI +3.2% to +10.8%, p<0.001), but specificity decreased from 84.5% to 78.9% (-5.5%, 95% CI -11.0% to -0.09%, p=0.046). Reader confidence AUC did not change significantly. ED clinicians with AI achieved sensitivity comparable to unaided radiologists, with no significant change in specificity. ConclusionAI-assisted interpretation increased reader sensitivity for critical abnormalities but reduced specificity. Notably, AI assistance enabled ED clinicians to reach diagnostic sensitivity similar to unaided radiologists, supporting the potential for AI to extend the diagnostic capabilities of non-radiologists. Further prospective studies are warranted to confirm these findings in real-world settings. FundingThis study was funded by Qure.ai via an NHSX Award EthicsThe study has been approved by the UK Healthcare Research Authority (IRAS 310995, approved 13/12/2022). The use of anonymised retrospective NCCTH has been authorised by Oxford University Hospitals. Trial registration numberNCT06018545. Research in contextO_ST_ABSWhat is already known on this topicC_ST_ABSAI-derived algorithms for the detection of pathological findings on non-contrast CT head (NCCTH) images have previously demonstrated strong diagnostic performance when used on retrospective datasets. AI-assisted image interpretation using these algorithms has been shown to enhance the diagnostic performance of general and neuro-radiologists in silico. The potential for AI to enhance the performance of less skilled readers who may encounter and be required to act on these images in clinical practice (e.g. non-specialist radiologists, emergency medicine clinicians and radiographers) is as yet untested, however. What this study addsThis large multicase multireader study demonstrates that AI-assisted image interpretation may be used to enhance the in silico diagnostic performance of Emergency Department physicians to a level comparable to that of general radiologists. How this study might affect research, practice or policyThis study raises the possibility that AI-assisted image interpretation could be used to assist non-radiologist clinicians in the safe interpretation of NCCTH scans. Further prospective research is required to test this hypothesis in clinical practice and explore the potential for AI-assisted interpretation to support safe discharge of patients with normal or low-risk scans.

Tags

Topics