Effect of an annotation-free artificial intelligence system for simultaneous detection and diagnosis on breast ultrasonography: a multireader, multicase study across diverse healthcare professionals.
Authors
Affiliations (7)
Affiliations (7)
- Department of Radiology, Severance Hospital, Research Institute of Radiological Science, Yonsei University College of Medicine, Seoul, Korea.
- Department of Radiology, Research Institute of Radiological Sciences, and Center for Clinical Imaging Data Science, Yonsei University College of Medicine, Seoul, Korea.
- Yonsei Institute for Digital Health, Yonsei University, Seoul, Korea.
- Department of Radiology, Wonkwang University Hospital, Wonkwang University School of Medicine, Iksan, Korea.
- Department of Radiology, School of Medicine, Kyungpook National University, Kyungpook National University Chilgok Hospital, Daegu, Korea.
- BeamWorks Inc., Daegu, Korea.
- School of Computer Science and Engineering, Kyungpook National University, Daegu, Korea.
Abstract
This study aimed to evaluate how a deep learning-based artificial intelligence (AI) system that provides automatic lesion detection and diagnosis affects the diagnostic performance of readers from diverse professional backgrounds. A total of 1,000 breast ultrasound (US) images (500 malignant, 250 benign, and 250 negative) were independently reviewed by 15 readers (six radiologists, three surgeons, three physicians, and three radiographers) in two sessions: session 1 without AI assistance and session 2 with AI assistance following a 2-week washout period. Reader performance across sessions was compared using the area under the localization receiver operating characteristic curve (AULROC), sensitivity, specificity, and accuracy. With AI assistance, the reader-averaged AULROC increased significantly from 0.864 (95% confidence interval [CI], 0.831 to 0.898) to 0.910 (95% CI, 0.888 to 0.931) (P=0.002). The standalone AI achieved an AULROC of 0.909 (95% CI, 0.889 to 0.930). Radiographers demonstrated a significant increase in AULROC, from 0.868 (95% CI, 0.840 to 0.897) to 0.922 (95% CI, 0.908 to 0.936) (P=0.032). Across all readers, sensitivity increased from 85.3% to 95.0% and accuracy from 82.3% to 85.7% (both P<0.001). Specificity did not change significantly among radiologists or surgeons (P=0.089 and P=0.955, respectively) but decreased among physicians (82.3% vs. 75.7%, P<0.001) and radiographers (81.7% vs. 76.7%, P<0.001). Radiologists without fellowship training or with <8 years of experience showed significantly improved sensitivity and accuracy, accompanied by decreased specificity after AI assistance (all P<0.05). AI assistance significantly increased the average diagnostic performance of readers in breast US interpretation. Among less experienced professionals and radiologists, improvements in sensitivity and accuracy were accompanied by decreased specificity.