Impact of Imaging Acquisition and Protocol Variability on Artificial Intelligence Model Performance: A Secondary Analysis of the ASFNR Artificial Intelligence Competition.
Authors
Affiliations (2)
Affiliations (2)
- From the Department of Neurology (G.Z., M.E.), The University of Arizona, Tucson, AZ, USA; Department of Diagnostic, Molecular and Interventional Radiology (BBO), Icahn School of Medicine at Mount Sinai, New York, NY, USA; Department of Radiology and Imaging Sciences (J.W.A.), Indiana University School of Medicine, Indianapolis, IN, USA; Department of Radiology (D.P.B.), Duke University Medical Center, Durham, NC, USA; Department of Radiology, Neuroradiology Division (R.C., A.C., S.H., B.J., G.Z.), Stanford University, Stanford, CA, USA; Sutter Imaging (R.C.), Sutter Health, Sacramento, CA, USA; Department of Neuroradiology (H.C.), MD Anderson Cancer Center, Houston, TX, USA; Department of Radiology (C.G.F.), Tufts University, Boston, MA, USA; Department of Radiology (A.E.F.), Thomas Jefferson University, Philadelphia, PA, USA; Department of Radiology (R.G.), University of Alabama at Birmingham, Birmingham, AL, USA; Department of Radiology & Biomedical Imaging (C.H.), University of California, San Francisco, San Francisco, CA, USA; Department of Radiology (K.H.), New York University Grossman School of Medicine, New York, NY, USA; Department of Radiology (J.A. M., S.S.N.), University of Texas Southwestern Medical Center, Dallas, TX, USA; Department of Clinical Neurosciences (P.M.), Lausanne University Hospital, Lausanne, Switzerland; Department of Diagnostic Radiology and Nuclear Medicine (P.R.), University of Maryland School of Medicine, Baltimore, MD, USA; The Russell H. Morgan Department of Radiology and Radiological Science (P.R.), Johns Hopkins University, Baltimore, MD, USA; The Malone Center for Engineering in Healthcare (H.I.S.), Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA; Department of Radiology (H.I.S.), Mayo Clinic, Rochester, MN, USA; and Department of Radiology (H.I.S.), Wake Forest University School of Medicine, Winston-Salem, NC, USA.
- From the Department of Neurology (G.Z., M.E.), The University of Arizona, Tucson, AZ, USA; Department of Diagnostic, Molecular and Interventional Radiology (BBO), Icahn School of Medicine at Mount Sinai, New York, NY, USA; Department of Radiology and Imaging Sciences (J.W.A.), Indiana University School of Medicine, Indianapolis, IN, USA; Department of Radiology (D.P.B.), Duke University Medical Center, Durham, NC, USA; Department of Radiology, Neuroradiology Division (R.C., A.C., S.H., B.J., G.Z.), Stanford University, Stanford, CA, USA; Sutter Imaging (R.C.), Sutter Health, Sacramento, CA, USA; Department of Neuroradiology (H.C.), MD Anderson Cancer Center, Houston, TX, USA; Department of Radiology (C.G.F.), Tufts University, Boston, MA, USA; Department of Radiology (A.E.F.), Thomas Jefferson University, Philadelphia, PA, USA; Department of Radiology (R.G.), University of Alabama at Birmingham, Birmingham, AL, USA; Department of Radiology & Biomedical Imaging (C.H.), University of California, San Francisco, San Francisco, CA, USA; Department of Radiology (K.H.), New York University Grossman School of Medicine, New York, NY, USA; Department of Radiology (J.A. M., S.S.N.), University of Texas Southwestern Medical Center, Dallas, TX, USA; Department of Clinical Neurosciences (P.M.), Lausanne University Hospital, Lausanne, Switzerland; Department of Diagnostic Radiology and Nuclear Medicine (P.R.), University of Maryland School of Medicine, Baltimore, MD, USA; The Russell H. Morgan Department of Radiology and Radiological Science (P.R.), Johns Hopkins University, Baltimore, MD, USA; The Malone Center for Engineering in Healthcare (H.I.S.), Whiting School of Engineering, Johns Hopkins University, Baltimore, MD, USA; Department of Radiology (H.I.S.), Mayo Clinic, Rochester, MN, USA; and Department of Radiology (H.I.S.), Wake Forest University School of Medicine, Winston-Salem, NC, USA. [email protected].
Abstract
Artificial intelligence (AI) models have shown promise in neuroradiology, yet their real-world generalizability remains uncertain, partly due to variability in imaging acquisition and protocols. We aimed to evaluate the impact of data source, scanner manufacturer, scan mode, slice thickness, and the AI models developed by participating teams on AI performance in this secondary analysis of the 2019 American Society of Functional Neuroradiology (ASFNR) AI Competition. We included 1,177 anonymized noncontrast head CT scans from five institutions. Four teams participated, developing models to detect acute ischemic stroke, intracranial hemorrhage, mass effect, and to assess age-appropriate normality. Generalized estimating equations (GEE) were used to evaluate the effects of the aforementioned variables on model performance, and collinearity diagnostics were applied to exclude redundant variables. Due to collinearity with scanner manufacturer, data source was excluded from the model. Across all tasks, the AI model employed significantly influenced performance. Scanner manufacturer was significantly associated with accuracy in detecting intracranial hemorrhage and acute ischemic stroke but not mass effect or age-based normality. Slice thickness significantly associated with detection of intracranial hemorrhage and mass effect, with thinner slices yielding higher accuracy, but showed no effect on ischemic stroke or normality assessments. Scan mode did not significantly influence performance for any task. This secondary analysis demonstrates that imaging acquisition and protocol variability may significantly affect AI model performance. Scanner manufacturer, slice thickness, and the developed AI model were significantly associated with model accuracy, whereas scan mode had no significant impact. Among these, the developed AI model consistently proved most influential, reflecting the importance of training data, model architecture, and preprocessing methods.