Multi-Domain Experiment Dataset for Evaluating Reproducibility Tools (E1-E20)

Overview

This repository provides a curated dataset designed to address the critical need for standardized benchmarks in evaluating reproducibility tools. By encompassing a diverse collection of computational experiments across multiple scientific disciplines, this dataset facilitates the assessment of existing reproducibility tools while identifying areas for improvement.

Dataset Composition

The dataset previously published includes 20 curated experiments (E1–E20) that successfully reproduced results using at least one reproducibility tool. These experiments span various research domains, reflecting the complexity and diversity of modern computational research:

• Computer Science: Experiments sourced from IEEE/ACM ICSE 2022 and VLDB 2021, covering topics such as software engineering, database management, and artificial intelligence.
• Life Sciences: Climate change studies.
• Health Sciences: Medical research.
• Social Sciences & Humanities: Economics and interdisciplinary computational research.

Each experiment is carefully documented, providing:

• The original source code and data (where available).
• Metadata including programming languages, project size, and dependencies.
• Execution instructions to facilitate reproducibility.

Purpose & Applications

This dataset serves as a benchmarking resource for researchers, developers, and tool creators working on reproducibility in computational science. It enables:

• Comparative evaluation of reproducibility tools by testing real-world computational experiments.
• Assessment of limitations in current platforms regarding dependency management, execution consistency, and software environment variability.
• Advancement of reproducibility standards by promoting transparency, reliability, and cross-disciplinary collaboration.

Reproducibility Tools Evaluated

The dataset has been used to assess the capabilities of eight major reproducibility tools, including:

• Whole Tale
• Code Ocean
• RenkuLab
• ReproZip
• Binder (No package available)
• Sciunit
• FLINC

The results highlight key challenges in modern computational reproducibility, including insufficient documentation, evolving software dependencies, and the need for more adaptable solutions.