There is a newer version of the record available.

Published June 5, 2026 | Version 1.1.6
Software Open

BenchChecker: Assessing the Credibility of Bug-Fixing Benchmarks for LLMs (Artifact)

Description

This artifact contains the BenchChecker implementation for artifact evaluation. The AE entry point is python -m benchchecker.cli full BENCHMARK CASE_ID TARGET_MODEL. The package loads benchmark metadata, clones the target repository into a workspace, runs Stage I repository-presence probing with real target-model responses, runs Stage II patch-presence checking with GumTree-backed AST patch localization, constructs replacement diffs when similarity is inconclusive, and executes containerized benchmark tests when configured. NiCad 6.2 is vendored under benchchecker/vendor/nicad/NiCad-6.2, and GumTree 4.0.0-beta4 is vendored under benchchecker/vendor/gumtree/gumtree-4.0.0-beta4. See docs/AE_QUICKSTART.md and docs/AE_USAGE.md for one-command instructions and environment requirements.

Notes

Version 1.1.6 refreshes the AE artifact packaging after reviewer audit: the source checkout now includes GumTree and FreeTXL, the Zenodo archive preserves Unix executable bits for GumTree and NiCad, standard Python distributions include the packaged data assets while excluding runtime staging files, and the documentation now states the Java runtime requirement for GumTree. It keeps the v1.1.5 GumTree-backed Stage II Algorithm 4 localization implementation.

Files

BenchChecker-AE-v1.1.6.zip

Files (76.3 MB)

Name Size Download all
md5:a7a0c523d63b251a550389dad5c58db6
76.3 MB Preview Download

Additional details

Related works

Is supplement to
https://anonymous.4open.science/r/BenchChecker-3CF2 (URL)