Can You Trust Your Own Eval? A Framework and Controlled Validation for Measuring Exploitability and Contamination in LLM Evaluation Harnesses

"Can You Trust Your Own Eval?" asks whether the evaluation-integrity failures recently demonstrated on public LLM benchmarks (exploit agents scoring 100% without solving any task, reward hacking, and defective graders) also affect the private "production" harnesses that gate enterprise model selection, procurement, and release decisions.

The paper contributes: (i) a threat model separating three failure families (contamination, exploitability, and defectiveness) mapped to the stages of a typical production harness; (ii) a taxonomy of harness vulnerability classes; (iii) a reproducible measurement protocol based on sealed-versus-leaky differential testing over open-weight models, with formal metrics for harness exploitability (HES), decision impact (Decision Flip Rate), and contamination (Perturbation Gap); and (iv) a sealed-harness reference design and audit checklist for practitioners.

In a fully reproducible validation over eight open-weight models and three harness families, a deliberately non-capable exploit captures all attainable score on the leaky code and multiple-choice harnesses (HES = 1.0) and half of it on a vulnerable judge, collapsing to ≤ 0.25 once each harness is sealed (0.06 on code, even against an adaptive attacker). A model fine-tuned on a leaked set shows a large perturbation gap (+0.78) that a clean model and a difficulty-matched held-out control do not. No proprietary data is required to reproduce or refute these claims. The work validates the instruments on controlled archetypes rather than auditing deployed harnesses; the external-validity step is left open.

This is a preprint (v1) and has not been peer-reviewed.

This record contains the paper (PDF) together with the full reproducibility package: source code, prompts, seeds, harness configurations, and the contaminated LoRA adapter configuration.