Comparative Evaluation of Proprietary and Open-Source Large Language Models for Systematic Multi-source Information Extraction in Interventional Oncology.
Authors
Affiliations (7)
Affiliations (7)
- Department of Diagnostic and Interventional Radiology, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Hugstetter Str. 55, 79106, Freiburg, Germany. [email protected].
- Department of Diagnostic and Interventional Radiology, Medical Center - University of Freiburg, Faculty of Medicine, University of Freiburg, Hugstetter Str. 55, 79106, Freiburg, Germany.
- Department of Diagnostic, Interventional and Pediatric Radiology, Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland.
- Department of Diagnostic and Interventional Radiology, Charité - Universitätsmedizin Berlin, corporate member of Freie Universität Berlin and Humboldt Universität zu Berlin, Berlin, Germany.
- Department of Diagnostic and Interventional Radiology, University Hospital Leipzig, Leipzig, Saxony, Germany.
- Department of Radiology, Klinikum rechts der Isar, Technical University of Munich (TUM), Munich, Germany.
- Department of Diagnostic and Interventional Radiology, University Hospital Tübingen, Tübingen, Germany.
Abstract
To compare proprietary (GPT-4o, Gemini 1.5 Pro) and open-source (Llama 3.1 70B, Llama 3.1 405B) large language models (LLMs) for extracting clinically relevant variables from transarterial chemoembolization (TACE) reports in patients with hepatocellular carcinoma (HCC). Retrospective analysis of 556 anonymized longitudinal TACE-related reports (radiology, interventional procedure, and clinical follow-up) from 50 patients with HCC treated between 2012 and 2024 at a single tertiary center was carried out. Models extracted predefined binary variables (e.g., modified Response Evaluation Criteria in Solid Tumors [mRECIST] tumor response, alpha-fetoprotein [AFP] dynamics, Barcelona Clinic Liver Cancer [BCLC] stage) and ordinal variables (e.g., liver segment involvement, vascular invasion, follow-up assessment) using a standardized system prompt and output template. Model performance was assessed by accuracy, ordinal scores, and longitudinal error rates using mixed-effects regression with patient-level random intercepts. Proprietary models outperformed open-source models. GPT-4o and Gemini achieved the highest mean accuracies for binary variables (0.87 ± 0.21 and 0.85 ± 0.16) and ordinal variables (4.15/5 and 4.10/5), significantly exceeding both Llama models (p < 0.05). GPT-4o showed the lowest longitudinal error rate for binary variables (0.01 vs 0.09-0.21 for the other models), indicating greater robustness over time. All models showed poor performance in vascular invasion detection and follow-up assessment. Proprietary LLMs can accurately extract most key TACE-related variables from routine clinical reports and may support decision-making in interventional oncology; however, all models showed poor performance in vascular invasion detection and follow-up assessment, so expert human oversight remains essential.