Radiology board-style exams and LLMs: a scoping review of model performance.
Authors
Affiliations (3)
Affiliations (3)
- NLP Section Chief at HT médica. Jaen, Spain. Electronic address: [email protected].
- Section Chief of Neuroradiology at HT médica. Jaen, Spain. Electronic address: [email protected].
- CMO at HT médica. Jaen, Spain. Electronic address: [email protected].
Abstract
Large Language Models (LLMs) are increasingly being evaluated for their ability to answer official radiology board-style examination questions. Understanding their accuracy, limitations, and potential applications in education is essential for assessing their utility in the field. A scoping review was conducted in October 2025 across PubMed, Scopus, and Web of Science, following PRISMA guidelines. Studies were included if they evaluated LLMs on official radiology board-style examination questions. After screening 205 unique records, 29 studies met the inclusion criteria. Data were extracted on study characteristics, including LLM type and version, input modality, language, examination type, answer format, comparison with humans, and reported outcomes. The reviewed studies evaluated multiple LLMs, predominantly GPT-based models (GPT-3.5, GPT-4, GPT-4 Turbo, GPT-4o), as well as Claude, Gemini, LLaMA 3, and Mixtral. Text-only evaluations generally yielded higher accuracy (≈65-90%) compared to multimodal tasks (45-89%). GPT-4 and its variants consistently outperformed earlier versions, occasionally exceeding average human performance. Open-source models such as LLaMA 3 70B and Mixtral achieved comparable results to proprietary models, offering advantages in local deployment and privacy. Few studies directly compared LLM performance with human radiologists. LLMs demonstrate promising performance in answering text-based radiology board-style exam questions, particularly GPT-4-based models. Nevertheless, significant limitations persist in multimodal tasks and complex reasoning scenarios.