ModelShare
Published July 26, 2024 | Version v2
Preprint Open

Ten Simple Rules for Good Model-Sharing Practices

Creators

  • 1. Kairoi
  • 2. SciLifeLab
  • 3. ROR icon University of California, San Diego
  • 4. ROR icon San Diego Supercomputer Center
  • 5. US-RSE
  • 6. ROR icon Arizona State University
  • 7. ROR icon Public Library of Science
  • 8. ROR icon Netherlands Institute of Ecology
  • 9. ROR icon Oak Ridge National Laboratory
  • 10. ROR icon Helmholtz-Zentrum Hereon
  • 11. Heliophysics Digital Resource Library
  • 12. NASA Goddard
  • 13. ROR icon Royal Netherlands Institute for Sea Research
  • 14. ROR icon European Molecular Biology Laboratory
  • 15. ROR icon Johns Hopkins University
  • 16. ROR icon University of Notre Dame
  • 17. ROR icon University of Oxford
  • 18. Linux Foundation
  • 19. Open Modeling Foundation (OMF)
  • 20. ROR icon DataCite
  • 21. ROR icon Nanjing Normal University
  • 22. ROR icon University of Virginia
  • 23. ROR icon North Carolina State University

Description

Computational models are complex scientific constructs that have become essential for us to better understand the world. To build on the findings of researchers across disciplines, models must be shared openly. However, there are no widely agreed-upon standards for sharing models. This paper suggests ten simple rules to both (i) ensure you share models in a way that is at least “good enough,” and (ii) enable others to lead the change towards better model-sharing practices.

Files

Ten Simple Rules for Good Model-Sharing Practices.pdf

Files (2.1 MB)

Additional details

Funding

Alfred P. Sloan Foundation
Community Standards for FAIR Scientific Computation: Establishing the Open Modeling Foundation G-2022-19471

References

  • Rosman T, Bosnjak M, Silber H, Koßmann J, Heycke T. Open science and public trust in science: Results from two studies, Public Understanding of Science, 2022;31(8): 1046-1062. doi: 10.1177/09636625221100686.
  • United Nations Educational, Scientific, and Cultural Organization, UNESCO Recommendation on Open Science, 2021. doi: 10.54677/MNMH8546.
  • Janssen MA, Pritchard C, Lee A. On code sharing and model documentation of published individual and agent-based models. Environ Model Softw. 2020;134;104873. doi: 10.1016/j.envsoft.2020.104873.
  • Kherroubi Garcia I. On the Ontology of Multidisciplinary Epistemic Groups. M.Sc. Thesis, London School of Economics and Political Science. 2022. doi: 10.5281/zenodo.7323712
  • Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data. 2016;3;160018. doi: 10.1038/sdata.2016.18.
  • Chue Hong NP, Katz DS, Barker M, Lamprecht AL, Martinez C, Psomopoulos FE, et al. RDA FAIR4RS WG. (2022). FAIR Principles for Research Software (FAIR4RS Principles). Zenodo. doi: 10.15497/RDA00068.
  • Psomopoulos F, Katz DS, Garijo D, Serrano-Solano B, Castro LJ, Fouilloux A, et al. FAIR for Machine Learning (FAIR4ML) IG rev-002. 2024. Available from: https://www.rd-alliance.org/rationale/fair-machine-learning-fair4ml-ig/rev-002/
  • OMF Certification Working Group videos [Internet]. Open Modeling Foundation;2024 Apr [cited 30 May 2024]. Available from: https://www.youtube.com/@OMFCWG/videos
  • Glavič P, Lukman R. Review of sustainability terms and their definitions. Journal of Cleaner Production. 2007;15(18): 1875-1885. doi: 10.1016/j.jclepro.2006.12.006.
  • Gest H. Evolution of Knowledge Encapsulated in Scientific Definitions. Perspectives in Biology and Medicine. 2001;44(4): 556-564. doi: 10.1353/pbm.2001.0063.
  • Curiel E. The many definitions of a black hole. Nature Astronomy. 2019;3: 27–34. doi: 10.1038/s41550-018-0602-1.
  • Research Data Alliance. Metadata Standards Catalog (v.2.1.). 2021. [cited 24 June 2024]. Available from: https://rdamsc.bath.ac.uk/subject-index
  • Calder M, Craig C, Culley D, de Cani R, Donnelly CA, Douglas R, et al. (2018) Computational modeling for decision-making: where, why, what, who and how. Royal Society Open Science. 2018. doi: 10.1098/rsos.172096.
  • Edmonds B, Le Page C, Bithell M, Chattoe-Brown E, Grimm V, Meyer R, Montañola-Sales C, Ormerod P, Root H, Squazzoni F. Different Modelling Purposes, Journal of Artificial Societies and Social Simulation, 2019;22(3)6. doi: 10.18564/jasss.3993.
  • Grimm V, Berger U, Bastiansen F, Eliassen S, Ginot V, Giske J, et al. A standard protocol for describing individual-based and agent-based models. Ecological Modeling. 2006;198(1-2): 115-126. doi: 10.1016/j.ecolmodel.2006.04.023.
  • Grimm V. Developing standards for modeling is critical and exciting: lessons from ODD and TRACE. ModelShare Workshops; Open Modeling Foundation; 2024 Feb 20. doi: 10.5281/zenodo.10684172.
  • Grimm V, Berger U, DeAngelis DL, Polhill JG, Giske J, Railsback SF. The ODD protocol: A review and first update, Ecological Modeling, 2010 Nov 24;211(23): 2760-2768. doi: 10.1016/j.ecolmodel.2010.08.019.
  • Grimm V, Railsback SF, Vincenot CE, Berger U, Gallagher C, DeAngelis DL. The ODD Protocol for Describing Agent-Based and Other Simulation Models: A Second Update to Improve Clarity, Replication, and Structural Realism. Journal of Artificial Societies and Social Simulation. 2020 Mar 31;23(2)7. doi: 10.18564/jasss.4259.
  • Carroll SR, Garba I, Figueroa-Rodríguez OL, Holbrook J, Lovett R, Materechera S, et al. The CARE Principles for Indigenous Data Governance. Data Science Journal. 2020;19(1):43. doi: 10.5334/dsj-2020-043.
  • FORRT. Our Community. 2024 [cited 15 June 2024]. In: FORRT [Internet]. Available from: https://forrt.org/about/community/.
  • 2i2c, The Carpentries, Center for Scientific Collaboration and Community Engagement, Invest in Open Infrastructure, MetaDocencia, Open Life Science. A Collaborative Interactive Computing Service Model for Global Communities. Zenodo. 2022 Aug 29. doi: 10.5281/zenodo.7025288.
  • Nicholas D, Boukacem-Zeghmouri C, Rodríguez-Bravo B, Watkinson A, Świgon M, Xu J, et al.. Early career researchers: observing how the new wave of researchers is changing the scholarly communications market. Revue française des sciences de l'information et de la communication. 2018 Jan 01. doi: 10.4000/rfsic.4635.
  • Nicholas D, Watkinson A, Boukacem-Zeghmouri C, Rodríguez-Bravo B, Xu J, Abrizah A, et al. Early career researchers: Scholarly behavior and the prospect of change. Learned Publishing. 2017;30: 157-166. doi: 10.1002/leap.1098.
  • Tennant JP, Waldner F, Jacques DC, Masuzzo P, Collister LB, Hartgerink CHJ. The academic, economic and societal impacts of Open Access: an evidence-based review. F1000Research. 2016;5:632. doi: 10.12688/f1000research.8460.3.
  • Open Modeling Foundation. Working Groups. 2024 [cited 03 July 2024]. In: Open Modeling Foundation [Internet]. Available from: https://www.openmodelingfoundation.org/governance/working-groups/#early-career-scholars-working-group
  • Meyer MF, Harlan ME, Hensley RT, Zhan Q, Barbosa CC, Börekçi NS, et al. Hacking Limnology Workshops and DSOS23: Growing a Workforce for the Nexus of Data Science. Open Science, and the Aquatic Sciences. Limnology and Oceanography Bulletin 2024;33: 35-38. doi: 10.1002/lob.10607.
  • Zollman KJ. The Credit Economy and the Economic Rationality of Science. The Journal of Philosophy. 2018;115: 5-33. doi: 10.5840/jphil201811511.
  • CoMSES Network. Computational Modeling Journals. 2024 [cited 30 June 2024]. In: CoMSES Net [Internet] Available from: https://www.comses.net/resources/journals/
  • Hettrick S. The Hidden REF: Celebrating everyone that makes research possible. ModelShare Workshops; Open Modeling Foundation; 2024 May 24. doi: 10.5281/zenodo.11266576
  • Brand A, Allen L, Altman M, Hlava M, Scott J. Beyond authorship: attribution, contribution, collaboration, and credit. Learned Publishing 2015;28: 151-155. doi: 10.1087/20150211.
  • All Contributors [Internet]. 2024 [cited 29 June 2024]. Available from: https://allcontributors.org/
  • Lumbard K, Ahuja V, Barron E, Foster D, Goggins S, Germonprez M. Types of contribution. 2023 [cited 30 June 2024]. In: CHAOSS [Internet]. Available from: https://chaoss.community/kb/metric-types-of-contributions/
  • Parsons MA, Katz DS, Langseth M, Ramapriyan H, Ramdeen S. Credit where credit is due. Eos. 2022;103. doi: 10.1029/2022EO220239.
  • Bladek M. DORA: San Francisco Declaration on Research Assessment (May 2013). College & Research Libraries News. 2014 Apr;75(4): 191-196. doi:10.5860/crln.75.4.9104.
  • Hettrick S. A not-so-brief history of Research Software Engineers. 2016 Aug 17 [cited 2024 May 30]. In: Software Sustainability Institute [Internet]. Available from: https://www.software.ac.uk/blog/not-so-brief-history-research-software-engineers-0
  • JuRSE. The worldwide RSE movement. 2024 [cited 2024 May 30]. In: JuRSE [Internet]. Available from: https://www.fz-juelich.de/en/rse/about-rse/rse-worldwide
  • Combemale B, Gray J. Rumpe B. Research software engineering and the importance of scientific models. Softw Syst Model. 2023 Jul 29;22: 1081–1083.doi: 10.1007/s10270-023-01119-z.
  • Han E. What is design thinking and why is it important? 2022 Jan 18 [cited 2024 Jun 26]. In: Harvard Business Review [Internet]. Available from: https://online.hbs.edu/blog/post/what-is-design-thinking
  • Parizi R, Prested M, Marczak S, Conte T. How has design thinking being used and integrated into software development activities? A systematic mapping. Journal of Systems and Software. 2022 May;187: 111217. doi: 10.1016/j.jss.2022.111217.
  • Staehelin D, Dolata M, Schwabe G. Managing Tensions in Research Consortia with Design Thinking Artifacts. In: Hehn J, Mendez D, Brenner W, Broy M, editors. Design Thinking for Software Engineering. Progress in IS. Springer Cham. doi: 10.1007/978-3-030-90594-1_9.
  • ModelShare Workshop | Research Computing [Internet]. Open Modeling Foundation;2024 Apr [cited 30 May 2024]. Video: 1:25:12. Available from: https://www.youtube.com/watch?v=wY2nqhtLsLo
  • Horsfall D, Cool J, Hettrick S, Oliveira Pisco A, Chue Hong NP, Haniffa M. Research software engineering accelerates the translation of biomedical research for health. Nat Med. 2023;29: 1313–1316. doi: 10.1038/s41591-023-02353-0.
  • Bennett A, Garside D, Praag CGv, Hostler TJ, Kherroubi Garcia I, Plomp E, et al. A Manifesto for Rewarding and Recognizing Team Infrastructure Roles. Journal of Trial & Error. 2023;4(1). doi: 10.36850/mr8.
  • Ringuette R, Murphy N, Petrenko M, Reardon K, Rigler J, Mays L, et al. Advocating for Equality of Contribution: The Research Software Engineer (RSE). Bulletin of the AAS. 2023;55(3). doi: 10.3847/25c2cfeb.3e6ae1d3.
  • Diehl P, da Silva R. Science Gateways: Accelerating Research and Education—Part I, Computing in Science & Engineering. 2023;25(1): 5-6. doi: 10.1109/MCSE.2023.3282517.
  • Barker M, Delgado Olabarriaga S, Wilkins-Diehr N, Gesing S, Katz DS, Shahand S, et al. The global impact of science gateways, virtual research environments and virtual laboratories, Future Generation Computer Systems. 201995: 240-248. doi: 10.1016/j.future.2018.12.026.
  • Rajesh Kalyanam, Lan Zhao, Carol Song, Larry Biehl, Derrick Kearney, I. Luk Kim, Jaewoo Shin, Nelson Villoria, and Venkatesh Merwade. 2019. MyGeoHub—A sustainable and evolving geospatial science gateway. Future Gener. Comput. Syst. 94, C (May 2019), 820–832. https://doi.org/10.1016/j.future.2018.02.005.
  • WaterHub. Research Highlights. 2024 [cited 2024 Jul 05]. In: MyGeoHub [Internet]. Available from: https://mygeohub.org/groups/water-hub
  • SWAT. Homepage. 2024 [cited 2024 Jul 05]. In: SWAT [Internet]. Available from: https://swat.tamu.edu/
  • Rajib MA, Merwade V, Kim IL, Zhao L, Song C, Zhe S. SWATShare – A web platform for collaborative research and education through online sharing, simulation and visualization of SWAT models. Environmental Modelling & Software, 2016 Jan;75: 498-512. doi: 10.1016/j.envsoft.2015.10.032.
  • U.S. Department of Agriculture. 2024 [cited 2024 Jul 05]. In: USDA [Internet]. Available from: https://www.usda.gov/
  • Spolsky J. The Joel Test: 12 Steps to Better Code. 2000 Aug 09 [cited 03 July 2024]. In: Joel on Software [Internet]. Available from: https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-steps-to-better-code/
  • Gass SI, Hoffman KL, Jackson RHF, Joel LS, Saunders PB. Documentation for a model: a hierarchical approach. Communications of the ACM. 1981;24(11): 728–733. doi: 10.1145/358790.358792.
  • Van Voorn GAK, Verburg RW, Kunseler E-M, Voder J, Janssen PHM. A checklist for model credibility, salience, and legitimacy to improve information transfer in environmental policy assessments. Environmental Modelling and Software. 2016;83: 224-236. doi: 10.1016/j.envsoft.2016.06.003.
  • Mitchell M, Wu S, Zaldivar A, Barnes P, Vasserman L, Hutchinson B. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* '19). Association for Computing Machinery. 220–229. doi: 10.1145/3287560.3287596.
  • White M, Haddad I, Osborne C, Yanglet XL, Abdelmonsef A, Varghese S. The Model Openness Framework: Promoting Completeness and Openness for Reproducibility, Transparency, and Usability in Artificial Intelligence. arXiv:2403.13784 [Preprint]. 2024 [cited 03 July 2024]. Available from: https://arxiv.org/abs/2403.13784 doi: 10.48550/arXiv.2403.13784.
  • Hadley L, Challenor P, Dent C, Isham V, Mollison D, Robertson DA, et al. Challenges on the interaction of models and policy for pandemic control. Epidemics. 2021;37:100499. doi: 10.1016/j.epidem.2021.100499.
  • Kee KF, Schrock AR. Best social and organizational practices of successful science gateways and cyberinfrastructure projects. Future Generation Computer Systems. 2018;94: 795-801. doi: 10.1016/j.future.2018.04.063.
  • Mao Y, Wang D, Muller M, Varshney KR, Baldini I, Dugan C, et al. How Data ScientistsWork Together With Domain Experts in Scientific Collaborations: To Find The Right Answer Or To Ask The Right Question? Proc. ACM Hum.-Comput. Interact. 2019;3. doi: 10.1145/3361118.
  • Radzvilavicius A. Computational reproducibility matters, Social Sciences. 2021 Sep 01 [cited 27 May 2024]. In: Springer Nature Research Communities [Internet]. Available from: https://communities.springernature.com/posts/computational-reproducibility-matters
  • Soiland-Reyes S, Sefton P, Crosas M, Castro LJ, Coppens F, Fernández JM, et al. Packaging research artefacts with RO-Crate. Data Science. 2022;5(2). doi: 10.3233/DS-210053
  • Harrison M. EMBL-EBIs approach to FAIR model sharing. ModelShare Workshops; Open Modeling Foundation; 2024 Apr 9. doi: 10.5281/zenodo.10877650.
  • NVIDIA, Vingelmann P, Fitzek FHP. CUDA, release: 10.2.89 [Internet]. 2020. Available from: https://developer.nvidia.com/cuda-toolkit
  • Godfrey MW. Understanding software artifact provenance. Science of Computer Programming. 2015;97(1): 86-90. doi: 10.1016/j.scico.2013.11.021.
  • Groth P, Jiang S, Miles S, Munroe S, Tan V, Tsasakou S., et al. An architecture for provenance systems, The PROVENANCE Consortium. 2006. [Cited 28 May 2024]. Available from: https://www.researchgate.net/publication/39994555_An_Architecture_for_Provenance_Systems
  • Walsh I, Fishman D, Garcia-Gasulla D, Titma T, Pollastri G, ELIXIR Machine Learning Focus Group, et al. DOME: recommendations for supervised machine learning validation in biology. Nature Methods. 2021;18: 1122–1127. doi: 10.1038/s41592-021-01205-4.
  • Cadwallader L, Mac Gabhann F, Papin J, Pitzer VE. Advancing code sharing in the computational biology community. PLoS Comput Biol 2022;18(6): e1010193. doi: https://doi.org/10.1371/journal.pcbi.1010193.
  • Janssen MA, Pritchard C, Lee A. On Code Sharing and Model Documentation of Published Individual and Agent-Based Models. Environmental Modelling & Software. 2020 Dec;134: 104873. doi: 10.1016/j.envsoft.2020.104873.
  • Buys M. Leveraging Connected Metadata to Support the Discoverability & Reuse of Open Models. ModelShare Workshops; Open Modeling Foundation; 2024 Apr 9. doi: 10.5281/zenodo.10961645.
  • Springer Nature. Springer Nature announces unified open code policy to better support open research practices. 2024 Feb 29 [cited 30 Jun 2024]. Available from: https://group.springernature.com/gp/group/media/press-releases/unified-code-sharing-policy-promoting-open-science/26789930
  • Pastrana E. Results from a Springer Nature-Code Ocean Pilot to Support Code Sharing. OSF [Preprint]. 2024 Mar 22 [cited 2024 03 Jul] Available from: https://osf.io/pzwfg/
  • Centola D, Becker J, Brackbill D, Baronchelli A. Experimental evidence for tipping points in social convention. Science. 2015;360: 1116-1119. doi: 10.1126/science.aas8827.
  • Cadwallader L. ModelShare workshop: Preservation and Publication - Perspectives from PLOS. ModelShare Workshops; Open Modeling Foundation; 2024 Mar 26. doi: 10.5281/zenodo.10877416.
  • Toribio-Flórez D, Anneser L, deOliveira-Lopes FN, Pallandt M, Tunn I, Windel H. Where Do Early Career Researchers Stand on Open Science Practices? A Survey Within the Max Planck Society. Frontiers in Research Metrics and Analytics. 2021 Jan 22;5:586992. doi: 10.3389/frma.2020.586992.
  • Gilbert N, Ahrweiler P, Barbrook-Johnson P, Preethi Narasimhan K, Wilkinson H. Computational Modelling of Public Policy: Reflections on Practice. Journal of Artificial Societies and Social Simulation. 2018;21(1)14. doi: 10.18564/jasss.3669.
  • Goodin D. Hugging Face, the GitHub of AI, hosted code that backdoored user devices. ArsTechnica. 2024 Mar 01 [cited 11 June 2024]. Available from: http://arstechnica.com/security/2024/03/hugging-face-the-github-of-ai-hosted-code-that-backdoored-user-devices
  • Yu Y, Yin G, Wang H, Wang T. Exploring the patterns of social behavior in GitHub. In Proceedings of the 1st International Workshop on Crowd-based Software Development Methods and Technologies (CrowdSoft 2014). Association for Computing Machinery. 2014: 31-36. doi: 10.1145/2666539.2666571.
  • Castaño J, Martínez-Fernández S, Franch X, Bogner J. Analyzing the Evolution and Maintenance of ML Models on Hugging Face. 2024 IEEE/ACM 21st International Conference on Mining Software Repositories (MSR), Lisbon, Portugal. 2024: 607-618
  • NASA Jet Propulsion Laboratory. Welcome to the HySDS Wiki. 2024 [cited 03 Jul 2024] Available from: https://hysds-core.atlassian.net/wiki/spaces/HYS/overview
  • Barnes N. Publish your computer code: it is good enough. Nature. 2010;467(753). doi: 10.1038/467753a.
  • Lemmen C, Sommer PS. Good Modeling Software Practices. arXiv:2405.21051 [Preprint]. 2024. doi: 10.48550/arXiv.2405.21051.
  • Ringuette R. Shaken not Stirred: Understanding and Preparing to Support Research Transparency through Publication Validation. Zenodo [Poster]. doi: 10.5281/zenodo.10934205.
  • European Parliament legislative resolution of 13 March 2024 on the proposal for a regulation of the European Parliament and of the Council on laying down harmonized rules on Artificial Intelligence (Artificial Intelligence Act) and amending certain Union Legislative Acts (COM(2021)0206 – C9-0146/2021 – 2021/0106(COD)). 2024 [cited 2024 Jun 23]. Available from: https://www.europarl.europa.eu/doceo/document/TA-9-2024-0138_EN.pdf
  • Executive Order No. 14110. 88 FR 75191. 2023 [cited 2024 Jun 23]. Available from: https://www.federalregister.gov/documents/2023/11/01/2023-24283/safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence
  • Wageningen Research Modeling Group. A quality checklist. Wageningen University & Research. 2023 [cited 2024 Jun 16]. Available from: https://magazines.wur.nl/kb-magazine-2023-en/a-quality-checklist
  • Houweling H, van Voorn GAK, van der Giessen A, Wiertz J. Quality of models for policy support. 2015 Jul [cited 2024 Jul 12]. Available from: https://edepot.wur.nl/362127
  • Hengeveld GM. A checklist for the quality of Models, Datasets and Indicators to be used in policy and decision support. Wageningen University & Research. 2020 [cited 2024 Jun 16]. Available from: https://assets.foleon.com/eu-central-1/de-uploads-7e3kk3/20634/wrqualitycriteriamodelsdatasets_2.d65c4f6abfac.pdf
  • Hengeveld GM, van der Greft-van Rossum JGM, de Bie PAF. Quality Assurance Models & Datasets WENR-WOT. 2021 Feb 24 [cited 2024 Jul 12]. Available from: https://edepot.wur.nl/542136
  • Annevelink EB, Meesters KPHK. Wikipedia for better computer models. 2023 Oct 09 [cited 2024 Jun 16]. In: Wageningen University & Research News [Internet]. Available from: https://www.wur.nl/en/newsarticle/wikipedia-for-better-computer-models.htm
  • Stopsack KHAB, Mucci LAA, Tworoger SSC, Kang JHD, Eliassen AHDE, Willett WCAE, et al. Promoting Reproducibility and Integrity in Observational Research: One Approach of an Epidemiology Research Community. Epidemiology. 2023;34(3): 389-395. doi: 10.1097/EDE.0000000000001599.
  • Science Mission Directorate. Scientific Information Policy for the Science Mission Directorate: SMD Policy Document SPD-41a. 2022 [cited 2024 June 23]. Available from: https://smd-cms.nasa.gov/wp-content/uploads/2023/08/smd-information-policy-spd-41a.pdf