Reproducing Deep Learning experiments: common challenges and recommendations for improvement

In computer science, there are more and more efforts to improve reproducibility. However, it is still difficult to reproduce the experiments of other scientists, and even more difficult when it comes to Deep Learning (DL). Making a DL research experiment reproducible requires a lot of work to document, verify, and make the system usable. These challenges are increased by the inherent complexity of DL, such as the number of (hyper)parameters, the huge amount of data, the versioning of the learning model, among others. Based on the reproduction of three DL case studies on real-world tasks, such as poverty estimation from remote sensing imagery, we identified common problems in the reproduction. Therefore, we proposed a set of recommendations ('fixes') to overcome these issues that a researcher may encounter in order to improve reproducibility and replicability and reduce the likelihood of wasted effort. These strategies can be used as "swiss army knife" to move from DL to more general areas as they are organized as (i) the quality of the dataset (and associated metadata), (ii) the Deep Learning method, (iii) the implementation, and the infrastructure used.

Poster to be presented during RDA 19th Plenary Meeting, Part Of International Data Week, 20–23 June 2022, Seoul, South Korea