Published November 4, 2026 | Version v9

Replication package for "When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench"

Authors/Creators

  • 1. Seoul School of Integrated Sciences and Technologies (aSSIST)

Description

Replication package for the manuscript "When Evaluators of AI Agents Disagree: An Empirical Study of Native Scoring, LLM Judges, and Commit-Boundary Checks on tau2-bench" (Sungbin Im, aSSIST; submitted to Empirical Software Engineering, November 2026).

Contents: the gate-injection harness and the four evaluator implementations (code/); all 2,160 schema-validated policy_event_trace records from tau2-bench airline and retail episodes with collision-free public identifiers (data/traces/); all evaluator outputs (native scoring, LLM judge, policy-aware judge, boundary evaluator), judge and gate prompts with model version identifiers, cost and judge ledgers, and derived analysis tables (data/analysis_out/); the annotation kit (rubric v1.3, blinded episode packets, both annotators' original label files under neutral names, recalibration records, adjudication worksheet with written justifications, final reference-standard labels) (annotation/); supplementary material S1–S4 and the claims table verified against the manuscript (docs/); and a manifest with SHA-256 checksums for every file. A top-level README describes the layout and the license (code MIT; data and documents CC BY 4.0).

Benchmark: sierra-research/tau2-bench at commit 363133a, used unmodified. Full simulation transcripts and the private packet-to-episode mapping are not included.

Files

artifact_v9.zip

Files (6.8 MB)

Name Size
md5:a56c77c621b771cac3c49f57f8a329ef
6.8 MB Preview Download

Additional details

Related works

Is supplement to
Preprint: arXiv:2611.14581 (arXiv)