Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results.
Authors
Affiliations (21)
Affiliations (21)
- Sycai Technologies SL, Scientific and Technical Department, Barcelona, Spain; BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain. Electronic address: [email protected].
- BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain.
- Sycai Technologies SL, Scientific and Technical Department, Barcelona, Spain.
- Universitätsklinikum Erlangen, Department of Radiology of the Uniklinikum Erlangen (UKER), Erlangen, Germany; University Hospital Erlangen, Imaging Science Institute, Erlangen, Germany.
- Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China; University of Chinese Academy of Sciences, Beijing, China.
- Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
- Friedrich-Alexander-Universität Erlangen-Nürnberg (FAU), Erlangen, Germany.
- Université de Rennes 1, CLCC Eugène Marquis, and INSERM UMR 1099 LTSI, Rennes, France.
- National Heart and Lung Institute, Faculty of Medicine, Imperial College London, London, United Kingdom.
- Centre for Medical Image Computing, Department of Computer Science, University College London, London, United Kingdom.
- Barcelona Artificial Intelligence in Medicine Lab (BCN-AIM), Facultat de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain.
- AIBA, Facultat de Matemàtiques i Informàtica, and Institute of Neuroscience, Universitat de Barcelona, Barcelona, Spain.
- Barcelona Artificial Intelligence in Medicine Lab (BCN-AIM), Facultat de Matemàtiques i Informàtica, Universitat de Barcelona, Barcelona, Spain; Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain.
- King's College London (KCL), London, United Kingdom.
- Universitat de Barcelona (UB), Barcelona, Spain.
- University of Edinburgh, Edinburgh, United Kingdom.
- Universitätsklinikum Erlangen, Department of Radiology of the Uniklinikum Erlangen (UKER), Erlangen, Germany.
- Hospital de Sant Pau i la Santa Creu, Diagnostic Imaging Department, Barcelona, Spain; Institut de Recerca Sant Pau - Centre CERCA, Advanced Medical Imaging, Artificial Intelligence, and Imaging-Guided Therapy Research Group, Barcelona, Spain.
- Hospital Universitari Vall d'Hebron, Department of Radiology, Institut de Diagnòstic per la Imatge (IDI), Barcelona, Spain.
- BCN Medtech, Universitat Pompeu Fabra, Barcelona, Spain; Institució Catalana de Recerca i Estudis Avançats (ICREA), Barcelona, Spain.
- TECNALIA, Basque Research and Technology Alliance (BRTA), Bizcaia, Spain.
Abstract
Deep learning (DL) has become the dominant approach for medical image segmentation, yet ensuring the reliability and clinical applicability of these models requires addressing key challenges such as annotation variability, calibration, and uncertainty estimation. This is why we created the Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS), which highlights the critical role of multiple annotators in establishing a more comprehensive ground truth, emphasizing that segmentation is inherently subjective and that leveraging inter-annotator variability is essential for robust model evaluation. Seven teams participated in the challenge, submitting a variety of DL models evaluated using metrics such as Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS). By incorporating consensus and dissensus ground truth, we assess how DL models handle uncertainty and whether their confidence estimates align with true segmentation performance. Our findings reinforce the importance of well-calibrated models, as better calibration is strongly correlated with the quality of the results. Furthermore, we demonstrate that segmentation models trained on diverse datasets and enriched with pre-trained knowledge exhibit greater robustness, particularly in cases deviating from standard anatomical structures. Notably, the best-performing models achieved high DSC and well-calibrated uncertainty estimates. This work underscores the need for multi-annotator ground truth, thorough calibration assessments, and uncertainty-aware evaluations to develop trustworthy and clinically reliable DL-based medical image segmentation models.