Evaluating Locally Deployed Large Language Models for Ga-68 PSMA PET/CT Report Mining in Prostate Cancer.
Authors
Affiliations (9)
Affiliations (9)
- Department of Nuclear Medicine, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
- SGTB Khalsa College, University of Delhi, New Delhi, India.
- Department of Urology, All India Institute of Medical Sciences, New Delhi, 110029 India.
- Department of Biomedical Engineering, Georgia Institute of Technology's, Wallace H. Coulter, Atlanta, GA 30332 USA.
- Department of Radiation Oncology, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
- Department of Radiodiagnosis & Interventional Radiology, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
- Department of Medical Oncology, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
- Department of Pathology, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
- Department of Biostatistics, All India Institute of Medical Sciences, New Delhi, Ansari Nagar, 110029 India.
Abstract
Large Language Models (LLMs) extracts structured data from unstructured Ga-68 Prostate Specific Membrane Antigen Positron Emission Tomography/Computed Tomography (PSMA PET/CT) reports by interpreting context and semantics, enabling reliable conversion of free-text into structured, queryable data. We evaluated whether locally deployed Llama 3.2:3b and Gemma 2:2b models could accurately perform zero-shot extraction of key diagnostic features. We retrospectively selected 50 de-identified Ga-68 PSMA PET/CT reports of histologically confirmed prostate cancer (01 January 2020-30 June 2024), ensuring rigorous expert annotation and inter-reader agreement. Reports were batch-processed using standardized zero-shot and few-shot prompts applied to locally deployed Llama 3.2:3b and Gemma 2:2b models, with model outputs benchmarked against an adjudicated dual-expert ground truth reference derived from independent annotations and consensus resolution of discrepancies using standard performance metrics. Both Llama 3.2:3b and Gemma 2:2b achieved rapid, reliable extraction of key diagnostic features from Ga-68 PSMA PET/CT reports, with excellent inter-reader agreement (average κ = 0.882). Llama 3.2:3b offered superior sensitivity (83.7%) and negative predictive value (NPV) (97.3%), while Gemma 2:2b demonstrated greater overall accuracy (86.2%) and specificity (88.5%). Model performance varied with question prevalence and complexity, highlighting distinct strengths for clinical integration. Open-weight Llama 3.2:3b and Gemma 2:2b LLMs enable rapid extraction of key PET/CT findings (16-20x faster than manual review), with Llama excelling in sensitivity and Gemma in specificity. However, modest performance on rare but clinically important findings and limited positive predictive value indicate these models are best suited for initial screening or human-assisted workflows rather than autonomous extraction. The online version contains supplementary material available at 10.1007/s13139-026-00991-z.