Open-source offline-deployable retrieval-augmented large language model for assisting pancreatic cancer staging.
Authors
Affiliations (7)
Affiliations (7)
- Department of Diagnostic Radiology, Faculty of Medicine, University of Yamanashi, 1110 Shimokato, Chuo, Yamanashi, Japan. [email protected].
- Department of Diagnostic Radiology, Faculty of Medicine, University of Yamanashi, 1110 Shimokato, Chuo, Yamanashi, Japan.
- Department of Therapeutic Radiology, Faculty of Medicine, University of Yamanashi, Yamanashi, Japan.
- Department of Radiation Oncology, Tohoku University Graduate School of Medicine, Sendai, Japan.
- Department of Internal Medicine, Kyonan Medical Center Fujikawa Hospital, Yamanashi, Japan.
- Division of Molecular Biology, Center for Medical Education and Sciences, Faculty of Medicine, University of Yamanashi, Yamanashi, Japan.
- Division of Mathematics, Center for Medical Education and Sciences, Faculty of Medicine, University of Yamanashi, Yamanashi, Japan.
Abstract
Large language models (LLMs) are increasingly applied in radiology, but key challenges remain, including data leakage from cloud-based systems, false outputs, and limited reasoning transparency. This study aimed to develop an open-source, offline-deployable retrieval-augmented LLM (RA-LLM) system in which local execution prevents data leakage and retrieval-augmented generation (RAG) improves output accuracy and transparency using reliable external knowledge (REK), demonstrated in pancreatic cancer staging. Llama-3.2 11B and Gemma-3 27B were used as local LLMs, and GPT-4o mini served as a cloud-based comparator. The Japanese pancreatic cancer guideline, written in English, served as REK. Relevant REK excerpts were retrieved to generate retrieval-augmented responses. System performance, including classification accuracy, retrieval metrics, and execution time, was evaluated on 100 simulated pancreatic cancer cases with CT findings described in English, using non-RAG LLMs as baselines. McNemar tests were applied to TNM staging and resectability classification. RAG improved TNM staging accuracy for all LLMs (GPT-4o mini 61% → 90%, p < 0.001; Llama-3.2 11B 53% → 72%, p < 0.001; Gemma-3 27B 59% → 87%, p < 0.001) and mildly improved resectability classification for GPT-4o mini and Llama-3.2 11B (72% → 84%, p = 0.012; 58% → 73%, p = 0.006), while the improvement for Gemma-3 27B was not evident (77% → 86%, p = 0.093). Gemma-3 27B showed performance comparable to GPT-4o mini. Retrieval performance was high (context recall = 1; context precision = 0.5-1), and local models ran at speeds comparable to the cloud-based GPT-4o mini. We developed an offline-deployable RA-LLM system for pancreatic cancer staging and publicly released its full source code. RA-LLMs outperformed baseline LLMs, and the offline-capable Gemma-3 27B performed comparably to the widely used cloud-based GPT-4o mini.