Published April 29, 2025 | Version v1
Report Open

Report on Challenges of Practical Reproducibility for Systems and HPC Computer Science

  • 1. ROR icon Argonne National Laboratory
  • 2. EDMO icon University of Chicago
  • 3. Instituto Universitario de Investigación en Ingenería de Aragón [I3A]
  • 4. ROR icon TU Wien
  • 5. ROR icon Sandia National Laboratories
  • 6. EDMO icon University of Missouri
  • 7. EDMO icon National Institute for Research in Computer and Control Sciences

Description

This report from the NSF-sponsored REPETO project synthesizes findings from the November 2024 Community Workshop on Practical Reproducibility in HPC, which convened researchers, artifact authors, reviewers, and chairs of reproducibility initiatives to address the critical challenge of making computational experiments reproducible in a cost-effective manner. The workshop deliberately focused on systems and HPC computer science research due to its unique requirements, including specialized hardware access and deep system reconfigurability. Through structured discussions, lightning talks, and panel sessions, participants identified key barriers to practical reproducibility and formulated actionable recommendations for the community.

The report presents a dual framework of challenges and recommendations organized by target audience (authors, reviewers, organizations, and community). It characterizes technical obstacles in experiment packaging and review, including completeness of artifact descriptions, acquisition of specialized hardware, and establishing reproducibility conditions. The recommendations range from immediate practical tools (comprehensive checklists for artifact packaging) to ecosystem-level improvements (refining badge systems, creating artifact digital libraries, and developing AI-assisted environment creation). Rather than advocating for reproducibility regardless of cost, the report emphasizes striking an appropriate balance between reproducibility rigor and practical feasibility, positioning reproducibility as an integral component of scientific exploration rather than a burdensome afterthought. Appendices provide detailed, immediately actionable checklists for authors and reviewers to improve reproducibility practices across the HPC community.

The workshop and report were supported by the NSF-sponsored REPETO project (award 2226406) and indirectly by the Chameleon project (NSF award 2027170).

FOR COMMENTS:

PLEASE SEND YOUR COMMENTS ON THE REPORT TO MTRICHARDSON@UCHICAGO.EDU.

FOR COMMENTS OR CONTRIBUTIONS TO THE CHECKLIST, PLEASE SEE OUR GITHUB.

Files

Reproducibility_Workshop_Report.pdf

Files (1.2 MB)

Name Size Download all
md5:1bd064e22a85a988fbdc691c5d69da1e
1.2 MB Preview Download

Additional details

Funding

U.S. National Science Foundation
Collaborative Research: Disciplinary Improvements: Repeto: Building a Network for Practical Reproducibility in Experimental Computer Science 2226406

Dates

Available
2025-04-29