There is a newer version of the record available.

Published August 13, 2026 | Version v1

Artifact deposit – Same Judgments, Different Conclusions: A Cluster-Aware Measurement Protocol for Pairwise LLM-as-a-Judge Evaluation in RAG

Authors/Creators

Description

Complete artifact deposit for the manuscript "Same Judgments, Different Conclusions: A Cluster-Aware Measurement Protocol for Pairwise LLM-as-a-Judge Evaluation in RAG", submitted for double-anonymized peer review. Author names, emails, institutions, and local paths have been replaced with [ANON-...] markers; no data values, verdicts, dates, or analysis content were altered.

The paper proposes a measurement protocol for pairwise LLM-as-a-judge evaluation of retrieval-augmented generation, built from four components: fixed evaluation budgets, cluster-aware inference, pre-registration with a public deviation log, and second-judge replication. The protocol is stress-tested in simulation, on a controlled two-domain RAG testbed of 400 multi-hop questions (computer science / machine learning and materials science), and on public MT-Bench human judgments.

This deposit contains every artifact behind every number in the paper:

- registrations/ - the audit trail in chronological order: the primary pre-registration, its addenda, the running deviation log, and one locked prediction plus its outcome document for each late experiment. Predictions were fixed before the corresponding analysis ran, and predictions that failed are reported verbatim rather than rewritten.
- judgments/ - per-item judge verdicts, one line per question, for every condition: the primary judge (Claude Opus 4.7) and the second-judge replications (DeepSeek V4 Pro).
- pairs/ - the judged material: each question, both answers, and the randomized A/B position mapping.
- results/ - every analysis output cited in the paper.
- scripts/ - all analysis, simulation, figure, and pipeline code.
- src/ - the evidence-selector package (BRKGA decoder, surrogate fitness, judge prompt).
- data/ - corpus manifests (arXiv identifiers with SHA-256 hashes), the 400 generated questions, and the public MT-Bench human judgments.
- figures_src/ - TikZ source of the pipeline diagram.
- logs/ - timestamped run logs with API costs, corroborating the registration chronology.

Reproduction. The README maps each number in the paper to the results file that holds it and the script that produces it. A single command (reproduce.sh, or the provided Dockerfile) regenerates the analyses and rebuilds the paper's figures offline, with no API keys and no network access: roughly two minutes for the analyses, fifteen with the simulation grids. Every result file that reproduce.sh regenerates is byte-identical to the shipped copy under fixed seeds. The README states which remaining files need inputs that are not redistributed here, and why.

Not included: the chunked corpus and embeddings (~300 MB), deterministically reconstructable from the arXiv identifiers and hashes in data/ via the included build scripts. arXiv content is referenced by identifier, not redistributed.

Files

deposit_anonymous.zip

Files (4.1 MB)

Name Size
md5:8b1a83fd373c25c26648f959b6341115
4.1 MB Preview Download

Additional details

Dates

Submitted
2026-08-13