Artificial intelligence and radiologists in pancreatic cancer detection using standard of care CT scans (PANORAMA): an international, paired, non-inferiority, confirmatory, observational study.
Authors
Affiliations (19)
Affiliations (19)
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands. Electronic address: [email protected].
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands.
- Department of Surgery and Oncology, Karolinska University Hospital, Stockholm, Sweden.
- Department of Quantitative Health Sciences, Cleveland Clinic, Cleveland, OH, USA.
- Department of Radiology, Haukeland University Hospital, Bergen, Norway.
- Department of Clinical Medicine, University of Bergen, Bergen, Norway.
- Department of Radiology, University Medical Center Groningen, Groningen, The Netherlands.
- Ziekenhuisgroep Twente, Almelo, The Netherlands.
- Department of Medical Imaging, Radboud University Medical Center, Nijmegen, The Netherlands.
- Department of Convergence Medicine, University of Ulsan College of Medicine, Asan Medical Center, Seoul, South Korea.
- Department of Radiology, Mayo Clinic, Rochester, MN, USA.
- Division of Medical Image Computing, German Cancer Research Center Heidelberg, Heidelberg, Germany.
- University of Bremen and Fraunhofer Institute for Medical Imaging MEVIS, Bremen, Germany.
- Institute of Applied Mathematical Sciences, National Taiwan University, Taipei, Taiwan.
- Department of Computer Science, Johns Hopkins University, Baltimore, MD, USA.
- Department of Radiology, Massachusetts General Hospital, Boston, MA, USA.
- Department of Radiology, Johns Hopkins Hospital, Baltimore, MD, USA.
- Department of Pathology, Institute of Clinical Medicine, University of Oslo, Oslo, Norway.
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands; Oncode Institute, Utrecht, the Netherlands.
Abstract
Pancreatic ductal adenocarcinoma (PDAC) has the worst prognosis among major cancer types, primarily due to late diagnosis on contrast-enhanced CT. Artificial intelligence (AI) can improve diagnostic performance, but robust benchmarks and reliable comparison to radiologists' performance are scarce. We established an open-source benchmark with the aim of investigating AI systems for PDAC detection on CT and compared them to radiologists' performance, at scale. In this international, paired, non-inferiority, confirmatory, observational study (PANORAMA), the AI system was trained and externally validated within an international benchmark, with a cohort of 2310 patients from four tertiary care centres in the Netherlands and the USA for training (n=2224) and tuning (n=86), and a sequestered cohort of 1130 patients from five tertiary care centres (the Netherlands, Sweden, and Norway) for testing. A multi-reader, multi-case observer study with 68 radiologists (40 centres, 12 countries; median 9·0 [IQR 6·0-14·5] years of experience) was conducted on a subset of 391 patients from the testing cohort. The reference standard was established with histopathology and at least 3 years of clinical follow-up. The primary endpoint was the mean area under the receiver operating characteristic curve (AUROC) of the AI system compared to that of radiologists at PDAC detection on CT. The study protocol and statistical plan were prespecified to test non-inferiority (considering a margin of 0·05), followed by superiority towards the AI system. This study is registered with Zenodo (https://doi.org/10.5281/zenodo.10599559) and is complete. Of the 3440 (1511 [44%] female, 1929 [56%] male; median age 67 [IQR 58-74] years) included patients (Jan 1, 2004 to Dec 31, 2023), 1103 (32%) received a positive PDAC diagnosis. In the sequestered testing cohort of 1130 patients (406 with histologically confirmed PDAC), AI achieved an AUROC of 0·92 (95% CI 0·90-0·93). In the subset of 391 patients (144 [37%] with histologically confirmed PDAC) used for the reader study, AI achieved statistically non-inferior (p<0·0001) and superior (p=0·001) performance with an AUROC of 0·92 (95% CI 0·89-0·94), compared to the pool of 68 participating radiologists, with an AUROC of 0·88 (0·85-0·91). AI demonstrated substantially improved PDAC detection on routine CT scans compared to radiologists on average, showing potential to detect cancer earlier and improve patient outcomes. European Union's Horizon 2020 research and innovation programme.