Comparing artificial intelligence and physician performance in predicting IDH mutation status in glioma.
Authors
Affiliations (38)
Affiliations (38)
- Division of Medical AI Research and Development, National Cancer Center Research Institute, Tokyo, Japan.
- AI Medical Engineering Team, RIKEN Center for Advanced Intelligence Project, Tokyo, Japan.
- Department of Neurosurgery and Neuro-Oncology, National Cancer Center Hospital, Tokyo, Japan. [email protected].
- Department of Neurosurgery, Tokai University School of Medicine, Isehara, Japan. [email protected].
- Department of Neurosurgery, Asahikawa Medical University, Asahikawa, Japan.
- Department of Diagnostic Radiology, National Cancer Center Hospital, Tokyo, Japan.
- Graduate School of Pharmaceutical Sciences, the University of Tokyo, Tokyo, Japan.
- Department of Neurosurgery, Graduate School of Medical Sciences, Kumamoto University, Kumamoto, Japan.
- Department of Neurosurgery, Kyorin University School of Medicine, Tokyo, Japan.
- Department of Neurosurgery, Dokkyo Medical University, Tochigi, Japan.
- Department of Neurosurgery, Tokyo Metropolitan Komagome Hospital, Tokyo, Japan.
- Department of Neurosurgery, Graduate School of Medicine, The University of Tokyo, Tokyo, Japan.
- Department of Neurological Surgery, Okayama University Graduate School of Medicine, Dentistry, and Pharmaceutical Sciences, Okayama, Japan.
- Department of Neurosurgery, Graduate School of Medical Sciences, Kyushu University, Fukuoka, Japan.
- Department of Neurosurgery, Institute of Science Tokyo, Tokyo, Japan.
- Department of Neurosurgery, Graduate School of Medicine, Yokohama City University, Yokohama, Japan.
- Department of Neuro-Oncology/Neurosurgery, Saitama Medical University International Medical Center, Saitama, Japan.
- Department of Neurosurgery, Osaka University Graduate School of Medicine, Suita, Japan.
- Department of Neurosurgery, Kansai Medical University, Hirakata, Japan.
- Department of Neurosurgery, NHO Osaka National Hospital, Osaka, Japan.
- Department of Neurosurgery, Osaka Metropolitan University Graduate School of Medicine, Osaka, Japan.
- Department of Neurological Surgery, Wakayama Medical University School of Medicine, Wakayama, Japan.
- Department of Neurosurgery, Osaka International Cancer Institute, Osaka, Japan.
- Department of Neurosurgery, Naniwaikuno Hospital, Osaka, Japan.
- Department of Biomedical Research and Innovation, Institute for Clinical Research, NHO Osaka National Hospital, Osaka, Japan.
- Department of Diagnostic Radiology, Hirosaki University Graduate School of Medicine, Hirosaki, Japan.
- Department of Radiology, Juntendo University School of Medicine, Tokyo, Japan.
- Department of Innovative Biomedical Visualization (iBMV), Graduate School of Medicine, Nagoya University, Nagoya, Japan.
- Department of Radiology, Shiga University of Medical Science, Otsu, Japan.
- Department of Radiology , Kyoto Prefectural University of Medicine, Kyoto, Japan.
- Department of Diagnostic Radiology, Kumamoto University Graduate School of Life Sciences, Kumamoto, Japan.
- Department of Radiology, Faculty of Medicine, University of Miyazaki, Miyazaki, Japan.
- Department of Radiology, Kagoshima University Graduate School of Medical and Dental Sciences, Kagoshima, Japan.
- Humanome Lab Inc., Tokyo, Japan.
- Department of Pathology, Kyorin University Faculty of Medicine, Tokyo, Japan.
- Department of Neurosurgery and Neuro-Oncology, National Cancer Center Hospital, Tokyo, Japan.
- Division of Medical AI Research and Development, National Cancer Center Research Institute, Tokyo, Japan. [email protected].
- AI Medical Engineering Team, RIKEN Center for Advanced Intelligence Project, Tokyo, Japan. [email protected].
Abstract
Predicting isocitrate dehydrogenase (IDH) mutations in gliomas using magnetic resonance imaging (MRI) is clinically important for treatment planning. This study compared two artificial intelligence (AI) models, GliomaDepth-IDH (ResNet34-based) and GliomaVista-IDH (Vision Transformer-based), with 18 physicians (eight neuroradiologists, five neurosurgeons, and five neurosurgery residents) in predicting IDH mutation status. On the Brain Tumor Segmentation Challenge dataset, the GliomaVista-IDH AI model achieved an area under the curve (AUC) value of 0.97, significantly outperforming all physician groups. However, external validation on a Japanese cohort revealed performance degradation: GliomaDepth-IDH declined to an AUC of 0.75 and GliomaVista-IDH to 0.82, with GliomaVista-IDH showing significant calibration issues (Brier score = 0.32). High-performing physicians achieved comparable results (AUC = 0.88) with superior calibration (Brier score = 0.19). Inter-rater reliability analysis revealed substantial variability across physician groups. These findings suggest that AI models can assist many physicians, while experienced practitioners remain competitive with better-calibrated predictions in challenging domains.