  Deep learning (DL) models have provided state-of-the-art performance in
various medical imaging benchmarking challenges, including the Brain Tumor
Segmentation (BraTS) challenges. However, the task of focal pathology
multi-compartment segmentation (e.g., tumor and lesion sub-regions) is
particularly challenging, and potential errors hinder translating DL models
into clinical workflows. Quantifying the reliability of DL model predictions in
the form of uncertainties could enable clinical review of the most uncertain
regions, thereby building trust and paving the way toward clinical translation.
Several uncertainty estimation methods have recently been introduced for DL
medical image segmentation tasks. Developing scores to evaluate and compare the
performance of uncertainty measures will assist the end-user in making more
informed decisions. In this study, we explore and evaluate a score developed
during the BraTS 2019 and BraTS 2020 task on uncertainty quantification
(QU-BraTS) and designed to assess and rank uncertainty estimates for brain
tumor multi-compartment segmentation. This score (1) rewards uncertainty
estimates that produce high confidence in correct assertions and those that
assign low confidence levels at incorrect assertions, and (2) penalizes
uncertainty measures that lead to a higher percentage of under-confident
correct assertions. We further benchmark the segmentation uncertainties
generated by 14 independent participating teams of QU-BraTS 2020, all of which
also participated in the main BraTS segmentation task. Overall, our findings
confirm the importance and complementary value that uncertainty estimates
provide to segmentation algorithms, highlighting the need for uncertainty
quantification in medical image analyses. Finally, in favor of transparency and
reproducibility, our evaluation code is made publicly available at:
https://github.com/RagMeh11/QU-BraTS.
