Reproducibility Scorecard to Assess Systems Biology Models

Systems biology modelling involves the mathematical representation of biological processes to study complex system behaviour and was expected to be less affected by the reproducibility crisis. However, models often fail to reproduce and the reasons for the failure and prevalence were not fully understood. We analyzed 455 kinetic models published in 152 peer-reviewed journals. Most of these models were manually encoded from scratch to assess reproducibility. Our investigation revealed that 49% of the models could not be reproduced using the information provided in the manuscripts. With further effort, an additional 12% could be reproduced either by empirical correction or support from authors. The other 37% remained non-reproducible due to missing parameter values, missing initial concentration, or inconsistent model structure. Among the corresponding authors, we contacted <30% responded. Models from many life science journals failed to reproduce, revealing a common problem in the peer-review process. Hence, we proposed a reproducibility scorecard to assess each model and address the reproducibility crisis. The aim of this scorecard is to help evaluate the reproducibility of the systems biology model and its simulation. It consists of eight questions with a unit score for each “yes” as an answer. The higher the total score, the more reproducible the model is expected to be. All eight questions may not always be applicable and hence, on the scale of 8, a minimum score of 4 is recommended. Model authors, reviewers, and journal editors can assess each systems biology model in a manuscript using this scorecard to support the peer-review process. For details see Tiwari et al., 2021 Mol Sys Bio https://doi.org/10.15252/msb.20209982 and https://www.ebi.ac.uk/biomodels/reproducibility.