Curated dataset of 18 computational experiments-(E1-E18)

Overview

This dataset, "Multi-Domain Experiment Dataset for Evaluating Reproducibility Tools (E1–E18)," is curated to facilitate the evaluation and benchmarking of reproducibility frameworks. It provides a structured and diverse collection of scientific experiments, enabling researchers and developers to test and compare different tools designed for computational reproducibility.

Dataset Composition

The dataset consists of 18 experiments (E1–E18) covering multiple scientific domains, including computer science, human-computer interaction (HCI), medicine, artificial intelligence, climate change, and economics. These experiments range from simple computational scripts to complex setups requiring integrated databases, multiple programming languages, and domain-specific computational environments.

Experiment Sources

To ensure a well-balanced dataset, experiments were sourced from peer-reviewed scientific conferences and open-access repositories:

• Computer Science:

  • Software Engineering: Experiments from the IEEE/ACM International Conference on Software Engineering (ICSE 2022), a premier venue for software engineering research.
  • Databases: Selected experiments from the International Conference on Very Large Databases (VLDB 2021), a leading database conference.
  • Human-Computer Interaction (HCI) and User Studies:
    • Experiments sourced from the European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE 2023).
    • These experiments focus on user studies related to software engineering and usability research.

• Interdisciplinary Fields (Collected from Zenodo):

  • Artificial Intelligence (AI): Experiments covering machine learning and AI-driven methodologies.
  • Climate Change: Computational experiments and simulations addressing environmental research.
  • Medicine: Experiments focusing on medical and healthcare applications.
  • Economics: Computational economic models and data analysis experiments.

• Related Work on Reproducibility Tools:

  • We included experiments previously used to evaluate reproducibility tools and methodologies.
  • This ensures alignment with prior research and enhances comparability across tools.
  • Among these, two key studies SciInc and SciUnit provided three fully documented experiments:
    • Chicago Food Inspections Evaluation;
    • Variable Infiltration Capacity;
    • Incremental Query Execution.

Methodology for Dataset Curation

To construct this dataset, we employed a structured selection process:

1. Scientific Conference Selection:

   • We identified key research areas within computer science and selected experiments from ICSE 2022, VLDB 2021, and ESEC/FSE 2023 (focusing on user studies in HCI).

2. Zenodo Repository Search:

   • Targeted searches were conducted using the keywords "Medical," "Artificial Intelligence," "Climate Change," and "Economics."
   • We filtered results to include only software repositories.
   • From the top 100 search results in each category, five experiments were randomly selected per domain.

3. Reproducibility Tools & Related Work:

   • We incorporated experiments previously used to evaluate existing reproducibility tools.
   • This selection ensures comparability and continuity with past reproducibility studies.