ACR's Assess-AI: A Registry for Real-World Performance Monitoring of Clinical Imaging AI.
Authors
Affiliations (7)
Affiliations (7)
- VP, Data Science and Informatics, American College of Radiology, Reston, VA. Electronic address: [email protected].
- American College of Radiology, Reston, VA.
- American College of Radiology Data Science Institute, Reston, VA; Chief Strategy Officer & Chief Medical Information Officer, HOPPR, Chicago, IL; MSK Radiologist, Palo Alto VA Medical Center, Palo Alto, CA.
- ACR Board of Chancellors; Chair ACR Commission on Informatics, Reston, VA, Professor of Radiology & SA Consultant Radiologist, Department of Radiology, Mayo Clinic, Rochester, MN.
- Data Science Institute, Reston, VA, Chief Data Science Officer, Vice Chairman of Radiology, Massachusetts General Hospital, Boston, MA.
- Vice Chair ACR Commission on Informatics, Reston, VA, Associate Professor and Vice Chair of Practice Transformation, Perelman School of Medicine at the University of Pennsylvania, Philadelphia, PA.
- Data Science Institute, Reston, VA, Senior Director, Mass General Brigham AI, Boston, MA.
Abstract
To describe the design, infrastructure, and functionality of the American College of Radiology's (ACR) Assess-AI registry, a national quality registry created to monitor the real-world performance of clinical imaging artificial intelligence (AI) models. Assess-AI is a registry within the National Radiology Data Registry (NRDR) that enables participating facilities to submit de-identified AI output, radiology report text, and DICOM study metadata through the ACR Connect platform. Data are normalized and compared with surrogate labels extracted from radiology reports using large language model (LLM)-based prompting pipelines. Concordance between AI outputs and extracted surrogate labels is computed centrally, with results delivered through interactive dashboards. Facilities may locally re-identify studies via the Forensics App for quality review. Assess-AI currently supports multiple imaging AI use cases including intracranial hemorrhage, pulmonary embolism, pneumothorax, large vessel occlusion, bone age, and cervical spine fracture. Participating facilities can visualize data completeness, monitor longitudinal concordance trends, compare performance with registry benchmarks, and explore discordance by demographic or technical factors. Local review workflows allow detailed evaluation of discordant cases, supporting root-cause analysis and AI governance. Assess-AI provides a scalable, privacy-preserving framework for post-deployment performance monitoring of imaging AI models. By combining standardized data ingestion, LLM-based surrogate labeling, interactive analytics, and local adjudication, the registry addresses critical gaps in evaluating real-world AI performance. Ongoing expansion will incorporate additional modalities, model types, and risk-adjusted benchmarking to further enhance clinical utility.