Reporting checklist for foundation and large language models in medical research (REFINE): an international consensus guideline.
Authors
Mese I,Akinci D'Antonoli T,Bluethgen C,Bressem K,Cuocolo R,Chaudhari A,Tejani AS,Isaac A,Ponsiglione A,Meddeb A,Khosravi B,Le Guellec B,Kahn CE,Suh CH,Pinto Dos Santos D,Koh DM,Tzanis E,Kotter E,Colak E,Kitamura F,Busch F,Nensa F,Yang G,Müller H,Kather JN,Nawabi J,Kleesiek J,Zhong J,Santinha J,Haubold J,de Almeida JG,Lekadir K,Marias K,Reiner LN,Maier-Hein L,Moy L,Adams LC,MartÃ-Bonmatà L,Paschali M,Moassefi M,Dietzel M,Huisman M,Ingrisch M,Klontzas ME,Papanikolaou N,Diaz O,Kuriki P,Seeböck P,Rouzrokh P,Strotzer QD,Park SH,Faghani S,Tayebi Arasteh S,Kim SH,Venugopal VK,Kim W,Kocak B
Affiliations (72)
Affiliations (72)
- Uskudar State Hospital, Department of Radiology, Istanbul, Türkiye.
- Division of Diagnostic and Interventional Neuroradiology, Department of Radiology, University Hospital Basel, Basel, Switzerland.
- University Children's Hospital Basel, Department of Pediatric Radiology, Basel, Switzerland.
- Institute for Diagnostic and Interventional Radiology, University Hospital Zurich, University of Zurich, Zurich, Switzerland.
- Stanford University, Department of Radiology, Stanford, United States of America.
- Technical University of Munich, School of Medicine and Health, Department of Diagnostic and Interventional Radiology, Klinikum rechts der Isar, TUM University Hospital, Munich, Germany.
- Technical University of Munich, School of Medicine and Health, Department of Cardiovascular Radiology and Nuclear Medicine, German Heart Center, TUM University Hospital, Munich, Germany.
- Department of Medicine, Surgery, and Dentistry, University of Salerno, Baronissi, Italy.
- Stanford University, Department of Biomedical Data Science, Stanford, United States of America.
- University of California, San Francisco, Department of Radiology and Biomedical Imaging, San Francisco, California, United States of America.
- King's College London, School of Biomedical Engineering and Imaging Sciences, LIHE, The London Institute for Healthcare Engineering, London, United Kingdom.
- Department of Advanced Biomedical Sciences, University of Naples "Federico II", Naples, Italy.
- Charité-Universitätsmedizin Berlin, Department of Neuroradiology, Humboldt-Universität zu Berlin, Freie Universität Berlin, Berlin Institute of Health, Berlin, Germany.
- Yale University, Yale School of Medicine, Department of Radiology and Biomedical Imaging, New Haven, Connecticut, United States of America.
- Mayo Clinic, Department of Radiology, Rochester, United States of America.
- Lille University Hospital, Department of Neuroradiology, Lille, France.
- University of Pennsylvania, Philadelphia, PA, United States of America.
- University of Ulsan College of Medicine, Department of Radiology and Research Institute of Radiology, Asan Medical Center, Seoul, Republic of Korea.
- University Medical Center Mainz, Department of Radiology, Mainz, Germany.
- Royal Marsden Hospital, Department of Radiology and AI Imaging Hub, Sutton, United Kingdom.
- Institute of Cancer Research, Division of Radiotherapy and Imaging, Sutton, United Kingdom.
- Artificial Intelligence and Translational Imaging (ATI) Lab, Department of Radiology, University of Crete School of Medicine, Heraklion, Greece.
- Medical Center - University of Freiburg Faculty of Medicine, Department of Diagnostic and Interventional Radiology, Freiburg, Germany.
- St. Michael's Hospital, Department of Medical Imaging, Unity Health Toronto, Toronto, Canada.
- University of Toronto Temerty Faculty of Medicine, Department of Medical Imaging, Toronto, Canada.
- Universidade Federal de São Paulo, Department of Diagnostic Imaging, São Paulo, Brazil.
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany.
- Shanghai Key Laboratory of Magnetic Resonance, Institute of Magnetic Resonance and Molecular Imaging in Medicine, East China Normal University, China.
- Informatics Institute, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland.
- University of Geneva, Geneva, Switzerland.
- The Sense Research and Innovation Center, Sion & Lausanne, Switzerland.
- Else Kroener Fresenius Center for Digital Health, Faculty of Medicine, TUD Dresden University of Technology, Dresden, Germany.
- Department of Medicine I, Faculty of Medicine, TUD Dresden University of Technology, Dresden, Germany.
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany.
- Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, United Kingdom.
- Department of Imaging, Tongren Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China.
- Shanghai Key Laboratory of Flexible Medical Robotics, Tongren Hospital, Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China.
- Digital Surgery Lab - Breast Cancer Research Program, Champalimaud Foundation, Lisbon, Portugal.
- University of Lisbon Faculty of Medicine, Department of Radiology, Lisbon, Portugal.
- Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
- Champalimaud Foundation, Lisbon, Portugal.
- Universitat de Barcelona, Departament de Matemà tiques i Informà tica, Barcelona, Spain.
- Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain.
- Hellenic Mediterranean University, Department of Electrical and Computer Engineering, Heraklion, Crete, Greece.
- Computational BioMedicine Laboratory, Institute of Computer Science, Foundation for Research and Technology (FORTH), Crete, Greece.
- German Cancer Research Center (DKFZ), Division of Intelligent Medical Systems, Heidelberg, Germany.
- National Center for Tumor Diseases (NCT), NCT Heidelberg, a partnership between DKFZ and University Hospital Heidelberg, Heidelberg, Germany.
- Heidelberg University Hospital, Surgical Clinic, Surgical AI Research Group, Heidelberg, Germany.
- Heidelberg University, Faculty of Mathematics and Computer Sciences, Heidelberg, Germany.
- Mohamed Bin Zayed University of Artificial Intelligence (MBZUAI), Abu Dhabi, United Arab Emirates.
- New York University Grossman School of Medicine, United States of America.
- Medical Imaging Department and Biomedical Imaging Research Group at Hospital Universitario y Politécnico La Fe and Health Research Institute, Valencia, Spain.
- Institute of Radiology, Universitätsklinikum Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany.
- Radboud University Medical Center, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands.
- LMU University Hospital, Department of Radiology, Munich, Germany.
- Munich Center for Machine Learning (MCML), Munich, Germany.
- relAI - Konrad Zuse School of Excellence in Reliable AI, Munich, Germany.
- Computational Clinical Imaging Group, Champalimaud Research, Lisbon, Portugal.
- AI Hub, Royal Marsden Hospital, Sutton, United Kingdom.
- UT Southwestern Medical Center, Department of Radiology, Dallas, TX, United States of America.
- Medical Anomaly Detection (MANO) Group, Computational Imaging Research (CIR), Department of Biomedical Imaging and Image-guided Therapy, Medical University of Vienna, Austria.
- Comprehensive Center for AI in Medicine, Medical University of Vienna, Vienna, Austria.
- University Hospital Regensburg, Department of Diagnostic and Interventional Radiology, Regensburg, Germany.
- University of Pennsylvania, Department of Radiology, Philadelphia, United States of America.
- Radiology Informatics Lab, Mayo Clinic, Department of Radiology, Rochester, United States of America.
- Lab for AI in Medicine, Department of Diagnostic and Interventional Radiology, University Hospital RWTH Aachen, Aachen, Germany.
- Stanford University School of Medicine, Department of Urology, Stanford, United States of America.
- Rajiv Gandhi Cancer Institute and Research Center, Department of Radiology, New Delhi, India.
- HOPPR, Illinois, United States of America.
- American College of Radiology Data Science Institute, Virginia, United States of America.
- Palo Alto VA Medical Center, California, United States of America.
- Basaksehir Cam and Sakura City Hospital, Department of Radiology, Istanbul, Türkiye.
Abstract
To develop the REporting checklist for FoundatIon and large laNguagE models (REFINE), an international reporting guideline for transparent and reproducible reporting of foundation model (FM) and large language model (LLM) studies in medical research, including imaging artificial intelligence (AI) applications. The protocol was prespecified and publicly archived. A modified Delphi process was conducted to establish reporting standards for unimodal and multimodal FM and LLM applications involving text, imaging, and structured data. The steering committee coordinated protocol development, expert recruitment, all Delphi rounds, and the harmonization phase. Decisions were made based on predefined consensus thresholds. In Rounds 1 and 2, structured ratings and free-text feedback informed iterative revisions. In the post-Delphi harmonization phase, terminology was standardized, and detailed reporting instructions were finalized. The REFINE development group comprised 57 contributors from 17 countries, and 54 panelists from 16 countries completed Rounds 1 and 2. The harmonization phase was completed by three expert panelists and the steering committee. The entire process produced a 44-item, six-section framework with standardized terminology and detailed reporting instructions, supported by an online platform for practical use (https://refinechecklist.github.io/refine/checklist.html). The REFINE provides a comprehensive, consensus-based reporting standard for medical FM and LLM research, including imaging AI studies. The online version facilitates practical implementation. The REFINE enables transparent, comparable, and reproducible reporting of FM and LLM studies, supporting reliable evidence synthesis in medical and imaging-focused AI studies.
Topics
Journal Article