Evaluation of Approved AI-based Brain Aneurysm Detection Software in Clinical Practice: Comparison with Radiologist Assessment and Image Re-review.

Authors

Abstract

This study evaluated the performance of artificial intelligence (AI)-based brain aneurysm detection software in clinical settings, aiming to assess its utility as a supportive tool for radiologists. Metrics included sensitivity, positive predictive value (PPV), F1 score, and false positives (FPs) per case. A retrospective analysis of 442 cases (March 2023-August 2024) compared AI detections against a reference standard derived from the radiologists' assessments and image re-review. Findings were categorized into true positives (TPs), FPs, and false negatives (FNs). Subgroup analyses covered aneurysm size, magnetic field strength of the MRI, patient age, and aneurysm location. The study included 442 cases (226 males, 216 females; median age 72). Out of 94 total aneurysms, the AI detected 73 TP and missed 21 FN. It also identified 520 FP. Overall, sensitivity was 77.7%, PPV was 12.3%, and the F1 score was 0.212. The FPs averaged 1.18 per case. Sensitivity varied by aneurysm size: 85.1% for ≤ 3 mm, 69.2% for 3-5 mm, and 50.0% for > 5 mm. Significant variability in FPs per case was observed across different magnetic field strengths. Performance also varied by patient age and aneurysm location. The AI software demonstrated moderate sensitivity, especially for smaller aneurysms. Variations in performance across different magnetic field strengths and aneurysm size suggest a need for more robust AI algorithms. Detailed analysis of aneurysm locations provides insights into areas where AI performance could be enhanced. Integrating the AI software as a supportive tool, combined with radiologist expertise, is hypothesized to enhance detection accuracy, though further studies are needed to quantify this combined effect.

Tags

Topics