A comparative study of four large language models in treatment decision-making for Hip impingement syndrome based on real-world data.

Authors

Abstract

Large language models (LLMs) hold potential value in medical decision support, yet empirical studies systematically comparing multiple models for femoroacetabular impingement (FAI) within Chinese clinical contexts remain scarce. This study aimed to compare the classification performance, decision consistency, and decision confidence of four mainstream LLMs (GPT-5, Gemini 2.5 Pro, DeepSeek-R1, Grok-4) in FAI treatment decision-making tasks for FAI based on real-world hospitalized cases, and to evaluate their clinical adaptability and limitations. We retrospectively included 26 hospitalized FAI cases from our institution (13 surgical group, 13 conservative group), all of which had ethically approved and informed consented. Case information was standardized into two input formats: structured radiology reports only (Group A), and structured radiology reports combined with structured medical records-referred to as multi-source structured text input in this study (Group B). Under both conditions, four LLMs were tasked with providing binary treatment decisions ("surgical" or "conservative") and indicating decision confidence on a scale from 0 to 100%. Primary evaluation metrics included accuracy, precision, sensitivity, specificity, F₁ score, and Cohen's kappa. Spearman correlation analysis was used to examine the relationship between LLM decision confidence and accuracy. A hierarchical analysis process (AHP) was employed to derive composite scores through multi-criteria weighting, enabling comprehensive comparison of LLM performance. Under Group B conditions (incorporating structured medical record information), GPT-5 demonstrated optimal performance, with accuracy of 88%, precision of 92%, sensitivity of 85%, specificity of 92%, F₁ score of 0.88, and kappa of 0.77, indicating high alignment between its decisions and real-world outcomes. Under the same conditions, the other models showed inferior performance (Gemini 2.5 Pro: 62% accuracy; DeepSeek-R1: 58%; Grok-4: 42%). Across both input modes, GPT-5 exhibited a significant positive correlation between decision confidence and accuracy (Group A: Spearman's <i>r</i> = 0.54; Group B: <i>r</i> = 0.55, both <i>P</i> < 0.01). The confidence-accuracy correlations for the other models were inconsistent and unstable. In the AHP-based composite scoring, GPT-5 achieved the highest Group B score (0.79) and ranked first overall. Overall results indicate that integrating structured radiology reports with structured medical record information (i.e., multi-source structured text information, which simulates a multimodal approach in this study) significantly enhances LLM performance in FAI treatment decision support tasks. In this exploratory, retrospective single-center study with a small sample size, GPT-5 outperformed the other evaluated models in FAI treatment decision support tasks based on Chinese structured clinical information and demonstrated, to some extent, calibrated decision confidence. These preliminary findings suggest the feasibility of LLMs in orthopedic decision support, however, validation in larger, multi-center, prospective studies is required before broader clinical application. Future studies should adopt multi-center, prospective designs to enhance evaluations of model reproducibility, interpretability, and clinical safety, thereby further validating broader applicability.

Tags

Topics