Comprehensive Performance Testing and External Validation of an AI Algorithm to Detect and Segment Brain Metastases.
Authors
Affiliations (7)
Affiliations (7)
- Department of Radiation Oncology, Miami Cancer Institute, Baptist Health South Florida, Miami, Florida, USA.
- Department of Oncological Sciences, Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA.
- ARTORG Center for Biomedical Engineering, University of Bern, Bern, Switzerland.
- Department of Radiation Oncology, Inselspital, Bern University and University of Bern, Bern, Switzerland.
- Division of Neurosurgery, Miami Neuroscience Institute, Baptist Health South Florida, Miami, Florida, USA.
- Department of Neuroscience, Herbert Wertheim College of Medicine, Florida International University, Miami, Florida, USA.
- Department of Medical Oncology, Miami Cancer Institute, Baptist Health South Florida, Miami, Florida, USA.
Abstract
Artificial intelligence (AI)-based models have shown initial promise in imaging brain metastasis; however many lack validation against advanced imaging-informed datasets, precluding external validity and limiting widespread adoption. To overcome these limitations, we performed comprehensive performance testing against reference standard metrics and externally validated an AI algorithm. As part of its FDA-clearance process, performance testing of a previously developed U-Net-based AI model was conducted on a multi-institutional cohort with reference standard established via consensus review by three neuroradiologists. External validation was performed on patients imaged with dual sequences (augmented) as well as an open-access dataset (UCSF-BMSR). Evaluation metrics included sensitivity, false positive (FP) rate, positive predictive value (PPV), Dice Similarity Coefficient (DSC), 95% Hausdorff distance (HD95), normalized surface distance (NSD), and qualitative physician assessment. In the FDA performance testing cohort, the AI algorithm achieved a sensitivity of 90.0% (95% CI: 87.0%-94.0%), DSC of 0.86 (95% CI: 0.83-0.89), and average FP rate of 0.57 lesions. In the augmented and open-access external validation cohort, a sensitivity of 81.4% (95% CI: 73.7%-89.1%) and 85.2% (95% CI: 83.0%-87.4%) with an average number of 0.22 and 1.19 FP lesions and DSCs of 0.70 (95% CI: 0.66-0.73) and 0.78 (95% CI: 0.77-0.78) were calculated, respectively. In the augmented external validation cohort, 46.3% of contours were rated as requiring major revisions. This AI algorithm demonstrated promising performance via three unique datasets. However, given the notable rate of contour revisions, these findings support its clinical role not as an autonomous system, but as a human-in-the-loop tool requiring physician oversight.