Can You Trust Your Own Eval? A Framework and Controlled Validation for Measuring Exploitability and Contamination in LLM Evaluation Harnesses

"Can You Trust Your Own Eval?" asks whether the evaluation-integrity failures recently demonstrated on public LLM benchmarks (exploit agents scoring 100% without solving any task, reward hacking, and defective graders) also affect the private "production" harnesses that gate enterprise model selection, procurement, and release decisions.

The paper contributes: (i) a threat model separating three failure families (contamination, exploitability, and defectiveness) mapped to the stages of a typical production harness; (ii) a taxonomy of harness vulnerability classes; (iii) a reproducible measurement protocol based on sealed-versus-leaky differential testing over open-weight models, with formal metrics for harness exploitability (HES), decision impact (Decision Flip Rate and an erased-distinction rate), and contamination (Perturbation Gap); and (iv) a sealed-harness reference design and audit checklist for practitioners.

In a fully reproducible validation over three harness families and up to seventeen open-weight models (0.5B–32B), a deliberately non-capable exploit captures all attainable score on the leaky code and multiple-choice harnesses and half of it on a vulnerable judge; sealing — enforced mechanically by output separation — drives it to the non-capable floor (0.06 easy tier / 0.10 hard tier on code, unchanged when the attacker adapts). A model fine-tuned on a leaked set shows a +0.78 perturbation gap that a clean model and a difficulty-matched control do not. Decision impact is defect-dependent: on a 53-problem, convention-mutated hard tier that removes the score ceiling, the weak-test defect flips zero of 89 decisively ordered model pairs (one-sided exact 95% bound 0.033 under pair independence; 0.16 under the most conservative model-level clustering) yet silently erases 18% of the sealed harness's decisive distinctions (38% at the easy tier's ceiling), while judge attacks and contamination change which candidate is selected. An external audit of lm-eval-harness's GSM8K extraction filters, reproduced verbatim, shows the same compression pattern on infrastructure we did not build: honest accuracy inflated by up to +0.105, with near-zero non-capable exploitability under either filter. No proprietary data is required to reproduce or refute these claims. The work validates the instruments on controlled archetypes rather than auditing deployed harnesses; the external-validity step is only begun by the lm-eval audit.

This is a preprint (v2) and has not been peer-reviewed. v2 adds the powered 17-model campaign on a de-saturated hard tier, the output-separated sealed grader with its released seal test, the erased-distinction companion metric, and the external lm-eval-harness audit.

This record contains the paper (PDF) together with the full reproducibility package: source code, prompts, seeds, harness configurations, all merged runs and reports (including the 17-model campaign with per-model provenance and pinned checkpoint digests), the contaminated LoRA adapter weights and training configuration, and the base-model provenance (Hugging Face repository and revision hash).