Automating the Referral of Bone Metastases Patients With and Without the Use of Large Language Models.
Authors
Affiliations (12)
Affiliations (12)
- Department of Neurological Surgery, NYU Langone Health, New York, New York, USA.
- New York University Grossman School of Medicine, New York, New York, USA.
- Department of Radiology, NYU Langone Health, New York, New York, USA.
- Department of Population Health, NYU Langone Health, New York, New York, USA.
- Division of Applied AI Technologies, NYU Langone Health, New York, New York, USA.
- Washington University School of Medicine, Saint Louis, Missouri, USA.
- Department of Orthopedic Surgery, NYU Langone Health, New York, New York, USA.
- Laura and Isaac Perlmutter Cancer Center, NYU Langone Health, New York, New York, USA.
- Department of Radiation Oncology, NYU Langone Health, New York, New York, USA.
- Department of Medicine, NYU Langone Health, New York, New York, USA.
- Center for Data Science, New York University, New York, New York, USA.
- Neuroscience Institute, NYU Langone Health, New York, New York, USA.
Abstract
Bone metastases, affecting more than 4.8% of patients with cancer annually, and particularly spinal metastases require urgent intervention to prevent neurological complications. However, the current process of manually reviewing radiological reports leads to potential delays in specialist referrals. We hypothesized that natural language processing (NLP) review of routine radiology reports could automate the referral process for timely multidisciplinary care of spinal metastases. We assessed 3 NLP models-a rule-based regular expression (RegEx) model, GPT-4, and a specialized Bidirectional Encoder Representations from Transformers (BERT) model (NYUTron)-for automated detection and referral of bone metastases. Study inclusion criteria targeted patients with active cancer diagnoses who underwent advanced imaging (computed tomography, MRI, or positron emission tomography) without previous specialist referral. We defined 2 separate tasks: task of identifying clinically significant bone metastatic terms (lexical detection), and identifying cases needing a specialist follow-up (clinical referral). Models were developed using 3754 hand-labeled advanced imaging studies in 2 phases: phase 1 focused on spine metastases, and phase 2 generalized to bone metastases. Standard McRae's line performance metrics were evaluated and compared across all stages and tasks. In the lexical detection, a simple RegEx achieved the highest performance (sensitivity 98.4%, specificity 97.6%, F1 = 0.965), followed by NYUTron (sensitivity 96.8%, specificity 89.9%, and F1 = 0.787). For the clinical referral task, RegEx also demonstrated superior performance (sensitivity 92.3%, specificity 87.5%, and F1 = 0.936), followed by a fine-tuned NYUTron model (sensitivity 90.0%, specificity 66.7%, and F1 = 0.750). An NLP-based automated referral system can accurately identify patients with bone metastases requiring specialist evaluation. A simple RegEx model excels in syntax-based identification and expert-informed rule generation for efficient referral patient recommendation in comparison with advanced NLP models. This system could significantly reduce missed follow-ups and enhance timely intervention for patients with bone metastases.