Federated Tumor Segmentation

International challenges have become the standard for validation of biomedical image analysis methods. We argue, though, that the actual performance even of the winning algorithms on “real-world” clinical data often remains unclear, as the data included in these challenges are usually acquired in very controlled settings at few institutions. The seemingly obvious solution of just collecting increasingly more data from more institutions in such challenges does not scale well due to privacy and ownership hurdles.

As the first challenge to ever be proposed for federated learning in medicine, the Federated Tumor Segmentation (FeTS) challenge 2021 intends to address these hurdles, both for the creation and the evaluation of tumor segmentation models. Specifically, the FeTS 2021 challenge uses clinically acquired, multi-institutional MRI scans from the BraTS challenge, as well as from various remote independent institutions included in the collaborative network of a real-world federation (www.fets.ai). The FeTS challenge focuses on the construction and evaluation of a consensus model for the segmentation of intrinsically heterogeneous (in appearance, shape, and histology) brain tumors, namely gliomas [1]. Compared to the BraTS challenge [2-4], the ultimate goal of FeTS is 1) the creation of a consensus segmentation model that has gained knowledge from data of multiple institutions without pooling their data together (i.e., by retaining the data within each institution), and 2) the evaluation of segmentation models in
such a federated configuration (i.e., in the wild).

The FeTS 2021 challenge is structured in two tasks:

• Task 1 ("Federated Training") aims at effective weight aggregation methods for the creation of a consensus model given a pre-defined segmentation algorithm for training, while also (optionally) accounting for network outages.
• Task 2 ("Federated Evaluation") aims at robust segmentation algorithms, given a pre-defined weight aggregation method, evaluated during the testing phase on unseen datasets from various remote independent institutions of the collaborative network of the fets.ai federation.

To prepare for both these tasks, the participants can use the information provided on data origin and acquisition settings during the training phase of the challenge.

We intend to add a third task in the FeTS challenge 2022 to account for adversaries during the training phase. The clinical relevance and importance of the FeTS challenge is that it addresses challenges related to privacy, legal, bureaucratic, and ownership concerns. Ground truth reference annotations are created and approved by expert neuroradiologists for every subject included in the training, validation, and testing datasets to quantitatively evaluate the performance of the participating algorithms.

Participants are free to choose whether they want to focus on only one or multiple tasks.

References

[1] M.J.Sheller, et al. "Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data." Scientific reports. 10:1-12, 2020. DOI: 10.1038/s41598-020-69250-1

[2] B. H. Menze, et al. "The Multimodal Brain Tumor Image Segmentation Benchmark (BRATS)", IEEE Transactions on Medical Imaging 34(10):1993-2024, 2015. DOI: 10.1109/TMI.2014.2377694

[3] S.Bakas, et al., “Identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the BRATS challenge”, arXiv preprint arXiv:1811.02629

[4] S. Bakas, et al., "Advancing The Cancer Genome Atlas glioma MRI collections with expert segmentation labels and radiomic features", Nature Scientific Data, 4:170117, 2017. DOI: 10.1038/sdata.2017.117

[5] T. Rohlfing, et al. The SRI24 multichannel atlas of normal adult human brain structure. Hum Brain Mapp. 31(5):798-819, 2010.

[6] Duan R, et al. PALM: Patient-centered Treatment Ranking via Large-scale Multivariate Network Meta-analysis. medRxiv. 2020.

[7] A. L. Simpson et al., “A large annotated medical image dataset for the development and evaluation of segmentation algorithms,” arXiv:1902.09063

[8] M. Wiesenfarth, et al. “Methods and open-source toolkit for analyzing and visualizing challenge results,” arXiv:1910.05121.

[9] L. Maier-Hein, et al., “Why rankings of biomedical image analysis competitions should be interpreted with care,” Nat. Commun., 9(1):1–13, 2018. DOI: 10.1038/s41467-018-07619-7

[10] R.Cox, et al. “A (Sort of) new image data format standard: NIfTI-1: WE 150”, Neuroimage, 22, 2004.

[11] S.Thakur, et al. “Brain Extraction on MRI Scans in Presence of Diffuse Glioma: Multi-institutional Performance Evaluation of Deep Learning Methods and Robust Modality-Agnostic Training”, NeuroImage, 220: 117081, 2020. DOI: 10.1016/j.neuroimage.2020.117081