Predicting Immunotherapy Response in Unresectable Hepatocellular Carcinoma: A Comparative Study of Large Language Models and Human Experts.
Authors
Affiliations (13)
Affiliations (13)
- Hefei Cancer Hospital of CAS, Institute of Health and Medical Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031, P. R. China.
- University of Science and Technology of China, Hefei, 230026, P. R. China.
- Department of Oncology, Hefei Cancer Hospital, Chinese Academy of Sciences, Hefei, 230031, P. R. China.
- Department of Interventional Radiology, Division of Life Sciences and Medicine, The First Affiliated Hospital of USTC, University of Science and Technology of China, Hefei, 230001, P. R. China.
- Department of Radiology, The First Affiliated Hospital of University of Science and Technology of China, Hefei, 230001, P. R. China.
- Department of Radiology, The First Affiliated Hospital of Anhui Medical University, Hefei, 230022, P. R. China.
- Department of Hepatobiliary Surgery, Division of Life Sciences and Medicine, Anhui Province Key Laboratory of Hepatopancreatobiliary Surgery, Anhui Provincial Clinical Research Center for Hepatobiliary Diseases, The First Affiliated Hospital of USTC, The University of Science and Technology of China, Hefei, 230001, P. R. China.
- Hefei Cancer Hospital of CAS, Institute of Health and Medical Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031, P. R. China. [email protected].
- University of Science and Technology of China, Hefei, 230026, P. R. China. [email protected].
- Department of Oncology, Hefei Cancer Hospital, Chinese Academy of Sciences, Hefei, 230031, P. R. China. [email protected].
- Hefei Cancer Hospital of CAS, Institute of Health and Medical Technology, Hefei Institutes of Physical Science, Chinese Academy of Sciences, Hefei, 230031, P. R. China. [email protected].
- University of Science and Technology of China, Hefei, 230026, P. R. China. [email protected].
- Department of Oncology, Hefei Cancer Hospital, Chinese Academy of Sciences, Hefei, 230031, P. R. China. [email protected].
Abstract
Hepatocellular carcinoma (HCC) is an aggressive cancer with limited biomarkers for predicting immunotherapy response. Recent advancements in large language models (LLMs) like GPT-4, GPT-4o, and Gemini offer the potential for enhancing clinical decision-making through multimodal data analysis. However, their effectiveness in predicting immunotherapy response, especially compared to human experts, remains unclear. This study assessed the performance of GPT-4, GPT-4o, and Gemini in predicting immunotherapy response in unresectable HCC, compared to radiologists and oncologists of varying expertise. A retrospective analysis of 186 patients with unresectable HCC utilized multimodal data (clinical and CT images). LLMs were evaluated with zero-shot prompting and two strategies: the 'voting method' and the 'OR rule method' for improved sensitivity. Performance metrics included accuracy, sensitivity, area under the curve (AUC), and agreement across LLMs and physicians.GPT-4o, using the 'OR rule method,' achieved 65% accuracy and 47% sensitivity, comparable to intermediate physicians but lower than senior physicians (accuracy: 72%, p = 0.045; sensitivity: 70%, p < 0.0001). Gemini-GPT, combining GPT-4, GPT-4o, and Gemini, achieved an AUC of 0.69, similar to senior physicians (AUC: 0.72, p = 0.35), with 68% accuracy, outperforming junior and intermediate physicians while remaining comparable to senior physicians (p = 0.78). However, its sensitivity (58%) was lower than senior physicians (p = 0.0097). LLMs demonstrated higher inter-model agreement (κ = 0.59-0.70) than inter-physician agreement, especially among junior physicians (κ = 0.15). This study highlights the potential of LLMs, particularly Gemini-GPT, as valuable tools in predicting immunotherapy response for HCC.