Utilizing GPT-4o with context-based approaches for improved query of ACR RADS guidelines.
Authors
Affiliations (3)
Affiliations (3)
- University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, Germany; University of Bonn, University Hospital Bonn, Quantitative Imaging Lab Bonn (QILaB), Germany.
- University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, Germany.
- University of Bonn, University Hospital Bonn, Clinic for Diagnostic and Interventional Radiology, Germany; University of Bonn, University Hospital Bonn, Quantitative Imaging Lab Bonn (QILaB), Germany. Electronic address: [email protected].
Abstract
Context-based large language models (LLM) with features like PDF integration and retrieval-augmented generation (RAG) may enhance accuracy on medical guideline queries. 200 representative questions were drafted across five major guidelines of the American College of Radiology (ACR) Reporting and Data Systems (RADS), with 40 questions each for CAD-RADS, BI-RADS, LI-RADS, PI-RADS, and Lung-RADS. Prompts were presented once to three GPT-4o-based models: without additional context, with PDF attachment of the guideline, and using web-based retrieval-augmented generation (RAG). GPT-4o with PDF support answered 90% of questions correctly (180/200), outperforming both GPT-4o with RAG (83%, 165/200; P < 0.001) and GPT-4o without additional context (70%, 140/200; P < 0.001). The average response time was similar across all models: GPT-4o with PDF (13.6 ± 6.8 sec), GPT-4o with RAG (13.5 ± 3.5 sec), and GPT-4o without context (13.6 ± 6.3 sec; P = 0.97). Incorporating contextual information such as guideline documents substantially improves the accuracy of GPT-4o in answering ACR RADS-related questions.