Human-in-the-loop validation of a sequential multi-LLM medical education pipeline.
Authors
Affiliations (16)
Affiliations (16)
- Department of Radiology, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea.
- Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea.
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea.
- Department of Radiology, Hallym University Dongtan Sacred Heart Hospital, Hwaseong, Republic of Korea.
- Department of Radiology, Seoul National University Bundang Hospital, Seongnam, Republic of Korea.
- Department of Radiology, Yeouido St. Mary's Hospital, College of Medicine, The Catholic University of Korea, Seoul, Republic of Korea.
- Department of Radiology, Kyung Hee University Hospital, Kyung Hee University College of Medicine, Seoul, Republic of Korea.
- Department of Radiology, Ansan Hospital, Korea University School of Medicine, Ansan, Republic of Korea.
- Centum Breast and Thyroid Clinic, Busan, Republic of Korea.
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea.
- Department of Radiology and the Research Institute of Radiological Science, Gangnam Severance Hospital, Yonsei University College of Medicine, Seoul, Republic of Korea.
- Department of Nuclear Medicine, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea.
- Department of Radiology, Samsung Medical Center, Sungkyunkwan University School of Medicine, Seoul, Republic of Korea.
- Department of Radiology, Samsung Changwon Hospital, Sungkyunkwan University School of Medicine, Changwon, Republic of Korea. [email protected].
- Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea. [email protected].
- Department of Radiology and Research Institute of Radiology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea. [email protected].
Abstract
Large language models (LLMs) can generate medical educational content at scale, but multi-dimensional human validation of multi-LLM pipelines has not been performed at scale. We evaluated a 7-stage sequential multi-LLM pipeline implemented exclusively with the Gemini family (Gemini 3 Flash and Pro for text; Gemini 3 Pro Image, the Nano Banana Pro preview, for images), producing 6000 flashcards and 833 infographics for radiology board preparation. Nine residents and eleven attending radiologists evaluated 1284 flashcards across 11 subspecialties in a two-phase design around fifth-stage (S5) feedback. Among 1100 evaluations of 980 text-level PASS cards, the evaluation-level false-negative rate (FNR) for blocking errors was 1.00% (95% CI 0.50-1.78%), exceeding the pre-specified 0.3% safety threshold; card-level (1.12%) and majority-vote (0.82%) analyses converged. S5 feedback was associated with increased blocking-error flags (39-54 evaluation-level observations, McNemar p = 0.003) and decreased technical accuracy and educational quality scores (both p < 0.001). In unadjusted analyses, attending radiologists identified more errors than residents (odds ratio 4.52, 95% CI 2.38-8.57), although workload-matched sensitivity analyses attenuated this gap. A preliminary image pilot of 19 items showed critical errors in both tables and infographics, with between-format comparisons underpowered and not significant after Holm correction; this image-related observation should be interpreted as exploratory. The conditional safety estimates, low-reliability reference standard, and rater- and workload-dependent patterns do not support fully automated deployment; automated feedback was associated with sharpened evaluator scrutiny, though the single-arm design cannot isolate useful cueing from anchoring or re-review effects.