SZZBench: Controlled Git Histories with Explicit Oracles for Diagnosing SZZ Implementations
Authors/Creators
Description
Overview
SZZBench is a reusable tool for generating and verifying controlled Git histories with explicit oracles and diagnosing SZZ implementations. Declarative YAML scenarios describe the histories, bug-fixing commits, required bug-inducing commits, and evaluation policies. SZZBench generates ordinary Git repositories, scores SZZ predictions through a common JSONL interface, and provides naive baselines and scenario-quality checks.
Contents
- 279 scenarios: 133 classic and 146 graph scenarios, organized into 17 categories.
- 233 control and 46 discriminating scenarios, together with 26 naive baselines.
- Source code, validators, deterministic repository generators, verifiers, evaluator, tests, and documentation.
- Adapters for four SZZ tools and recorded predictions and evaluation reports for nine configurations.
- Sources for a local scenario editor and generated-repository viewer.
- Release provenance, a SHA-256 manifest, and a validation report.
Source snapshot
Repository: https://github.com/higolab/SZZBench
Commit: fb5091bab24afd06564939ab9d73c40ff5a978b2
Snapshot identifier: 2026-10-27-fb5091b. The Python package declares version 0.1.0. All 753 tracked source files are preserved byte for byte; four supplementary release files identify and document this snapshot.
Requirements and quickstart
Python 3.10 or later and Git 2.29 or later are required. The core Python dependency is PyYAML. From the extracted SZZBench directory on macOS or Linux:
python3 -m venv .venv source .venv/bin/activate python -m pip install -e . szzbench generate-all-repos szzbench evaluate --predictions examples/pydriller-run/predictions.jsonl --include-generated --out results/pydriller-recorded szzbench baselines --out results/baselines
The recorded PyDriller run should produce PASS 161, FAIL 118, precision 0.781 and recall 0.689. Recorded predictions can be re-scored without installing the third-party SZZ implementations. Start with ZENODO_README.md in the archive for complete instructions.
Fresh execution and reuse
To execute an SZZ implementation anew, follow the README for its adapter under examples/. Fresh PyDriller execution uses PyDriller 2.12. PySZZ, SZZ Unleashed, modified cregit and their dependencies remain external; their sources and Docker images are not bundled. The token-level Docker build does not fully pin its base image and apt packages. The web tools require Node.js and npm dependencies; some tests require a JDK. Generated Git repositories can be recreated from the included scenarios. Git history, local experiment outputs, virtual environments and node_modules are excluded.
Validation of this snapshot
All 279 scenario declarations passed simulation and lint with zero errors or warnings. All 279 Git repositories were generated and verified with zero errors. The verifier emitted 34 warnings: 20 basic-blame mismatches with the declared oracles, 13 unavailable basic-blame checks, and one nonunique fix-statement match. These diagnostics are documented in VALIDATION.json; strict generation treats the warnings as failures.
342 targeted tests passed with zero failures or skips. All nine recorded configurations were re-scored on newly generated repositories and matched their archived aggregate metrics and per-scenario outcomes. The 26 baselines and coverage commands completed successfully. Fresh third-party SZZ execution, web-tool builds and the full test suite were outside this release check.
Scope and licenses
The controlled synthetic histories support diagnosis and regression testing. They do not estimate accuracy or the frequency of failure conditions in real projects. The scenarios are predominantly Java, and Windows has not been verified.
Software is licensed under the MIT License. Scenarios, examples and Markdown documentation are licensed under CC BY 4.0. Both license texts and LICENSES.md, which defines their scope and attribution requirements, are included.
Files
SZZBench-0.1.0-fb5091b.zip
Additional details
Software
- Repository URL
- https://github.com/higolab/SZZBench