AI in Action: A Roadmap from the Radiology AI Council for Effective Model Evaluation and Deployment.
Authors
Affiliations (2)
Affiliations (2)
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA. Electronic address: [email protected].
- Department of Radiology and Imaging Sciences, Emory University, Atlanta, GA, USA.
Abstract
As the integration of artificial intelligence (AI) into radiology workflows continues to evolve, establishing standardized processes for the evaluation and deployment of AI models is crucial to ensure success. This paper outlines the creation of a Radiology AI Council at a large academic center and subsequent development of framework in the form of a rubric to formalize the evaluation of radiology AI models and onboard them into clinical workflows. The rubric aims to address the challenges faced during the deployment of AI models, such as real-world model performance, workflow implementation, resource allocation, return on investment (ROI), and impact to the broader health system. Using this comprehensive rubric, the council aims to ensure that the process for selecting AI models is both standardized and transparent. This paper outlines the steps taken to establish this rubric, its components, and initial results from evaluation of 13 models over an 8-month period. We emphasize the importance of holistic model evaluation beyond performance metrics, and transparency and objectivity in AI model evaluation with the goal of improving the efficacy and safety of AI models in radiology.