Poster Open Access

Reproducing Deep Learning experiments: common challenges and recommendations for improvement

Jeaneth Machicao; Ali Ben Abbes; Leonardo Meneguzzi; Pedro Pizzigatti Corrêa; Alison Specht; Romain David; Gérard Subsol; Danton Ferreira Vellenich; Rodolphe Devillers; Shelley Stall; Nicolas Mouquet; Marc Chaumont; Laure Berti-Equille; David Mouillot

In computer science, there are more and more efforts to improve reproducibility. However, it is still difficult to reproduce the experiments of other scientists, and even more difficult when it comes to Deep Learning (DL). Making a DL research experiment reproducible requires a lot of work to document, verify, and make the system usable. These challenges are increased by the inherent complexity of DL, such as the number of (hyper)parameters, the huge amount of data, the versioning of the learning model, among others. Based on the reproduction of three DL case studies on real-world tasks, such as poverty estimation from remote sensing imagery, we identified common problems in the reproduction. Therefore, we proposed a set of recommendations ('fixes') to overcome these issues that a researcher may encounter in order to improve reproducibility and replicability and reduce the likelihood of wasted effort. These strategies can be used as "swiss army knife" to move from DL to more general areas as they are organized as (i) the quality of the dataset (and associated metadata), (ii) the Deep Learning method, (iii) the implementation, and the infrastructure used.

Poster to be presented during RDA 19th Plenary Meeting, Part Of International Data Week, 20–23 June 2022, Seoul, South Korea

Acknowledgments: The PARSEC project is funded by the Belmont Forum, Collaborative Research Action on Science-Driven e-Infrastructures Innovation. J.M. is grateful for the support from FAPESP (grant 2020/03514–9).
176
101
views
downloads
All versions This version
Views 176176
Downloads 101101
Data volume 103.3 MB103.3 MB
Unique views 139139
Unique downloads 8080

Share

Cite as