Clinical Usability of Generative Artificial Intelligence for MR Safety Advice

Authors

Abstract

This study investigated whether readily available, generative AI models, could be used to answer MR safety queries as an MR Safety Expert (MRSE), with "clinical usability" assessed by an expert review panel. This study is a mixed retrospective-prospective, proof-of-concept study. A clinical MR safety advice archive (January 2024 to April 2025) was used to curate 30 generic MR safety support requests with associated MRSE responses. ChatGPT-4o (ChatGPT) and Google AI Overview (GAIO) were prompted with these generic requests to generate AI safety advice. An expert panel assessed all answers for clinical usability. Unusable responses were assigned as; "Unsafe Advice", "Safe but Incorrect", "Incomplete Advice/ Key Details Missing", "Contradictory Statements"," Out of Date". Requests were subcategorised into "specific" and "generic" requests, as well as "passive" and "active" implants, and "other" requests for post review analysis. Percentages of usable answers and reasons for non-usable responses were compared. Overall, 93% (28/30) of the human responses, 50% (15/30) of the GAIO responses and 43% (13/30) of the ChatGPT responses were deemed acceptable for clinical use. Subcategorization usability was: "generic"; Human 94% (16/17), GAIO and ChatGPT 59% (10/17), "specific": Human 92% (12/13), GAIO 38% (5/13), ChatGPT 23% (3/13), Active: Human 100% (9/9), GAIO 33% (3/9), ChatGPT 22% (2/9), "passive": Human 88% (14/16), GAIO 56% (9/16), ChatGPT 50% (8/16) and "other"; Human 100% (5/5), GAIO and ChatGPT 60% (3/5). While both AIs were able provide clinically acceptable answers for some requests they did so at a significantly lower success rate than a human MRSE.

Tags

Topics