Exploring the Role of AI in Enhancing Nuclear Medicine Report Impressions Generated by Trainees and ChatGPT-4o: Comparative Evaluation Study.
Authors
Affiliations (2)
Affiliations (2)
- Division of Nuclear Medicine and Molecular Imaging, Department of Medicine, The Ottawa Hospital, University of Ottawa, 501 Smyth Road, Ottawa, ON, K1H 8L6, Canada, 1 613-798-5555 ext 71278, 1 613-737-8752.
- Department of Medical Imaging, Division of Nuclear Medicine, McGill University Health Center, McGill University, Montreal, QC, Canada.
Abstract
Accurate reporting in nuclear medicine is essential for clinical decision-making. Trainees often generate preliminary reports with variable quality, and artificial intelligence (AI) tools such as ChatGPT-4o may enhance report clarity and accuracy, particularly in the impression section of the report. This study aimed to evaluate and compare the quality of positron emission tomography / computed tomography report impression sections generated by trainees and by the AI chatbot ChatGPT-4o (OpenAI), focusing on correctness, clarity, completeness, organization, use of diagnostic certainty terminology, and physician satisfaction. The impression sections of 200 positron emission tomography/ computed tomography reports generated by trainees and AI (100 reports each) were compared. The AI generated the impressions based on the stem, clinical history, and technical sections of the trainee-generated reports. Reports were blindly rated by 3 nuclear medicine physicians. Survey questions, including Likert-scale questions, assessed correctness, use of certainty terms, clarity, completeness, organization, and satisfaction. Statistical analyses included Fisher exact test, 2-tailed t tests, chi-square tests, ANOVA, and effect sizes (Cohen d). Thematic analysis was performed on free-text comments. AI-generated impressions were rated as correct in 92% of reports (mean 0.92, SD 0.27) and trainee-generated impressions rated as correct in 91% of reports (mean 0.91, SD 0.29; P=.80, negligible effect size). AI-generated impressions included certainty terms more frequently than trainee-generated impressions (94% vs 81%, respectively; P=.005; χ²2=6.58, small effect size). AI also used higher-level certainty terms more frequently than trainees (mean 4.23, SD 1.25 vs mean 3.69, SD 1.93 on a 5-point scale; P=.02). Clarity was high in both groups (AI: mean 4.49, SD 0.70, and 89.8% of reports rated as clear vs trainees: mean 4.35, SD 0.85, and 87.0% of reports rated as clear; P=.20). Completeness was significantly higher for AI-generated impressions than trainee-generated impressions (mean 4.52, SD 0.76, and 90.4% complete vs mean 4.09, SD 1.01, and 81.8% complete, respectively; P<.001, small-to-medium effect size). Organization was similar between the groups (mean 4.22, SD 0.91 vs mean 4.39, SD 0.74; P=.15). Satisfaction ratings were also compared, with AI-generated impressions achieving a mean score of 4.31 (SD 0.77) and 86.2% satisfaction, compared with a mean score of 4.14 (SD 0.91) and 82.8% satisfaction for trainee-generated impressions (P=.16). Thematic analysis showed that trainee-generated impressions were more frequently criticized for accuracy (74% vs 19%; P<.001) and actionability (20% vs 4.8%; P=.02), whereas AI-generated impressions were more frequently criticized for excessive length (33% vs 0%; P<.001). These findings suggest that the AI chatbot ChatGPT-4o can serve as a valuable adjunct in nuclear medicine reporting, particularly at the resident level, by improving decisiveness and completeness while complementing trainee education and clinical oversight.