Published May 13, 2026
| Version v1.0.0
Software
Open
ReproScore: Separating Readiness from Outcome in Research Software Reproducibility Assessment
Authors/Creators
- 1. FIZ Karlsruhe - Leibniz-Institut fur Informationsinfrastruktur GmbH Berlin
-
2.
Freie Universität Berlin
-
3.
Leibniz Institute of Freshwater Ecology and Inland Fisheries
Description
Separating Readiness from Outcome in Research Software Reproducibility Assessment.
ReproScore is a two-tier scoring framework for assessing the reproducibility of research software repositories. It separates reproducibility readiness (what a repository contains) from reproducibility outcome (whether the software actually runs), making the distinction explicit and measurable.
Overview
| Score | What it measures | When available |
|---|---|---|
| RRS | Static readiness — 26 sub-metrics across 5 categories | Always (no execution needed) |
| ROS | Execution outcome — up to 6 sandboxed probes | When execution infrastructure is available |
| RCS | Coverage-adaptive composite of RRS + ROS | When any ROS component is available |
The core insight is that a repository can score well on static readiness yet fail to execute (e.g. pinned dependencies with a version conflict), and conversely execute successfully despite minimal static signals. ReproScore makes this readiness–outcome gap visible rather than conflating the two quantities.
Repository structure
reproscore/
├── ablation_analysis.py ← Reproduce all evaluation statistics from scores.csv
├── config/
│ └── default_rubric.yaml ← Category weights (w_i), gate parameters (τ, k), penalties
├── src/
│ ├── scoring/
│ │ ├── rrs.py ← 26 sub-metric detectors, gate function, category aggregation
│ │ ├── ros.py ← Execution outcome scoring (6 components)
│ │ ├── rcs.py ← Composite score with coverage weight α
│ │ └── rubric.py ← YAML rubric loader and validator
│ └── utils/
│ └── notebook_paths.py ← Notebook discovery and exclusion filters
├── tests/
│ └── test_scoring.py ← Unit tests for RRS (no execution required)
└── data/
└── ablation/
└── 20260511_101920/ ← Evaluation run (423 repositories)
├── scores.csv ← Per-repository RRS scores + 26 sub-metrics + ground truth
├── analysis_results.json
├── analysis.log
├── provenance.json
├── logs/ ← Clone and score logs
└── repos/ ← Per-repository score provenance (JSON, one file per repo)
Quick start
Install
git clone https://github.com/myVSR/reproscore.git
cd reproscore
pip install -r requirements.txt
Score a repository
from src.scoring.rrs import RRSScorer
result = RRSScorer().score("/path/to/repo")
print(f"RRS: {result.rrs:.1f}")
for sym, cat in result.category_scores.items():
print(f" {sym}: {cat.raw_score:.1f}")
for ev in result.evidence:
if ev.fix_suggestion:
print(f" [{ev.metric_id}] {ev.fix_suggestion}")
Reproduce evaluation statistics
python ablation_analysis.py
# or point at the bundled run explicitly:
python ablation_analysis.py --run-dir data/ablation/20260511_101920
Run tests
pytest tests/test_scoring.py -v
Notes
If you use this software or dataset, please cite it using the metadata below.
Files
myVSR/reproscore-v1.0.0.zip
Additional details
Related works
- Is supplement to
- Software: https://github.com/myVSR/reproscore/tree/v1.0.0 (URL)
Software
- Repository URL
- https://github.com/myVSR/reproscore
- Programming language
- Python