Published November 15, 2026
| Version v3
Software
Open
Artifact deposit – Same Judgments, Different Conclusions: A Cluster-Aware Measurement Protocol for Pairwise LLM-as-a-Judge Evaluation in RAG
Authors/Creators
Description
Complete artifact deposit for the manuscript "Same Judgments, Different Conclusions: A Cluster-Aware Measurement Protocol for Pairwise LLM-as-a-Judge Evaluation in RAG", submitted for double-anonymized peer review. Author names, emails, institutions, and local paths have been replaced with [ANON-...] markers; no data values, verdicts, dates, or analysis content were altered. The paper proposes a measurement protocol for pairwise LLM-as-a-judge evaluation of retrieval-augmented generation, built from four components: fixed evaluation budgets, cluster-aware inference, pre-registration with a public deviation log, and second-judge replication. The protocol is stress-tested in simulation, on a controlled two-domain RAG testbed of 400 multi-hop questions (computer science / machine learning and materials science), and on public MT-Bench human judgments. This deposit contains every artifact behind every number in the paper: - registrations/ - the audit trail in chronological order: the primary pre-registration, its addenda, the running deviation log, and one locked prediction plus its outcome document for each late experiment. Predictions were fixed before the corresponding analysis ran, and predictions that failed are reported verbatim rather than rewritten. - judgments/ - per-item judge verdicts, one line per question, for every condition: the primary judge (Claude Opus 4.7) and the second-judge replications (DeepSeek V4 Pro). - pairs/ - the judged material: each question, both answers, and the randomized A/B position mapping. - results/ - every analysis output cited in the paper. - scripts/ - all analysis, simulation, figure, and pipeline code. - src/ - the evidence-selector package (BRKGA decoder, surrogate fitness, judge prompt). - data/ - corpus manifests (arXiv identifiers with SHA-256 hashes), the 400 generated questions, and the public MT-Bench human judgments. - figures_src/ - TikZ source of the pipeline diagram and of the validity-chain figure. - docs/human_eval_qval/ - the blinded human audit of question validity: the annotation interface as the annotators used it (index.html), the instructions, the three annotation files, and the key that maps each audit item to its question and stratum. - logs/ - timestamped run logs with API costs, corroborating the registration chronology. Version 3 (second revision) adds the clustering diagnostic for Testbed 1 (scripts/testbed1_cluster_diagnostic.py) and the test-inverted intervals for every comparison in the forest figure (scripts/forest_inverted_intervals.py), and updates the validity-chain figure source. Reproduction. The README maps each number in the paper to the results file that holds it and the script that produces it. A single command (reproduce.sh, or the provided Dockerfile) regenerates the analyses and rebuilds the paper's figures offline, with no API keys and no network access: roughly two minutes for the analyses, fifteen with the simulation grids. Every result file that reproduce.sh regenerates is byte-identical to the shipped copy under fixed seeds. The README states which remaining files need inputs that are not redistributed here, and why. Not included: the chunked corpus and embeddings (~300 MB), deterministically reconstructable from the arXiv identifiers and hashes in data/ via the included build scripts. arXiv content is referenced by identifier, not redistributed.
Files
GADMEC_IPM_deposit_anonymous_v3.zip
Additional details
Dates
- Submitted
-
2026-08-13