Published October 27, 2026 | Version 2026-10-27-fb5091b

SZZBench: Controlled Git Histories with Explicit Oracles for Diagnosing SZZ Implementations

  • 1. ROR icon The University of Osaka
  • 2. ROR icon Kyushu University

Description

Overview

SZZBench is a reusable tool for generating and verifying controlled Git histories with explicit oracles and diagnosing SZZ implementations. Declarative YAML scenarios describe the histories, bug-fixing commits, required bug-inducing commits, and evaluation policies. SZZBench generates ordinary Git repositories, scores SZZ predictions through a common JSONL interface, and provides naive baselines and scenario-quality checks.

Contents

  • 279 scenarios: 133 classic and 146 graph scenarios, organized into 17 categories.
  • 233 control and 46 discriminating scenarios, together with 26 naive baselines.
  • Source code, validators, deterministic repository generators, verifiers, evaluator, tests, and documentation.
  • Adapters for four SZZ tools and recorded predictions and evaluation reports for nine configurations.
  • Sources for a local scenario editor and generated-repository viewer.
  • Release provenance, a SHA-256 manifest, and a validation report.

Source snapshot

Repository: https://github.com/higolab/SZZBench
Commit: fb5091bab24afd06564939ab9d73c40ff5a978b2
Snapshot identifier: 2026-10-27-fb5091b. The Python package declares version 0.1.0. All 753 tracked source files are preserved byte for byte; four supplementary release files identify and document this snapshot.

Requirements and quickstart

Python 3.10 or later and Git 2.29 or later are required. The core Python dependency is PyYAML. From the extracted SZZBench directory on macOS or Linux:

python3 -m venv .venv
source .venv/bin/activate
python -m pip install -e .
szzbench generate-all-repos
szzbench evaluate --predictions examples/pydriller-run/predictions.jsonl --include-generated --out results/pydriller-recorded
szzbench baselines --out results/baselines

The recorded PyDriller run should produce PASS 161, FAIL 118, precision 0.781 and recall 0.689. Recorded predictions can be re-scored without installing the third-party SZZ implementations. Start with ZENODO_README.md in the archive for complete instructions.

Fresh execution and reuse

To execute an SZZ implementation anew, follow the README for its adapter under examples/. Fresh PyDriller execution uses PyDriller 2.12. PySZZ, SZZ Unleashed, modified cregit and their dependencies remain external; their sources and Docker images are not bundled. The token-level Docker build does not fully pin its base image and apt packages. The web tools require Node.js and npm dependencies; some tests require a JDK. Generated Git repositories can be recreated from the included scenarios. Git history, local experiment outputs, virtual environments and node_modules are excluded.

Validation of this snapshot

All 279 scenario declarations passed simulation and lint with zero errors or warnings. All 279 Git repositories were generated and verified with zero errors. The verifier emitted 34 warnings: 20 basic-blame mismatches with the declared oracles, 13 unavailable basic-blame checks, and one nonunique fix-statement match. These diagnostics are documented in VALIDATION.json; strict generation treats the warnings as failures.

342 targeted tests passed with zero failures or skips. All nine recorded configurations were re-scored on newly generated repositories and matched their archived aggregate metrics and per-scenario outcomes. The 26 baselines and coverage commands completed successfully. Fresh third-party SZZ execution, web-tool builds and the full test suite were outside this release check.

Scope and licenses

The controlled synthetic histories support diagnosis and regression testing. They do not estimate accuracy or the frequency of failure conditions in real projects. The scenarios are predominantly Java, and Windows has not been verified.

Software is licensed under the MIT License. Scenarios, examples and Markdown documentation are licensed under CC BY 4.0. Both license texts and LICENSES.md, which defines their scope and attribution requirements, are included.

Files

SZZBench-0.1.0-fb5091b.zip

Files (2.1 MB)

Name Size
md5:41f5cbc76216e7b29bfb429006157883
2.1 MB Preview Download
md5:276ddc4a4a6bdfe1327b634abbdfe300
93 Bytes Download

Additional details

Software