Reproducing the predictions of a super-resolution adversarial model: a case study

We present the results of a case study on the reproducibility of an adversarial deep learning approach to super resolve wind and solar data [Stengel, et. al. 2020].  In this study, with the original code and data available, we trained the model 1,000 times on the same system under various experimental conditions: 1) using the same parameters as the original study, 2) varying the number of epochs, 3) varying learning rates, and 4) varying both the number of epochs and the learning rate. Our simulations show that the performance evaluation results based on the relative root mean square error (RRMSE) metric display substantial variability that may not be fully explained by the initial random seeds over the 1,000 runs. Our case study points to the need to report not only the performance but also its reproducibility when evaluating deep learning models, to have a more comprehensive picture of their predictive capacity and potential limitations.