Visual Oracle Bench (Phase 1, v1.6) — Two-Judge Synthetic-HTML Pilot for LLM-as-Judge Visual Regression Detection with Specificity Reporting

Phase 1 methodological-pilot snapshot for the EMSE Methodological Articles submission, release tag v1.6.1-emse-option-b. Supersedes v0.3.0; see the Data Integrity narrative in the manuscript (§1.4, §3.5, §6.1) for the silent-fabrication discovery and correction that distinguishes v1.6 from v0.3.0 headline numbers.

Pre-registration: OSF DOI 10.17605/OSF.IO/NKD6J, registered 2026-06-06 (Phase 2 design). Phase 1 Amendment 1 filed 2026-06-11.

Contains: 600-pair specificity-inclusive corpus (manifest v2: 400 defects + 200 controls); patched judge wrappers with malformed propagation; dispatcher fail-fast gate; analyzer with the _flag_silent_fabrications retroactive filter; paired-test script; v1.6 manuscript and cover letter; the contaminated original parquets retained for audit reproducibility.

Headline numbers (v1.6, integrity-audited subset): OpenAI gpt-5-codex 87.5% accuracy / 73.1% recall / 100% specificity (n=375); Claude Sonnet 4.5 78.4% accuracy / 45.8% recall / 100% specificity (n=208, attrition-confined to two of eight profiles); pairwise Cohen's κ = 0.679 [0.565, 0.788] on the n=208 mixed intersection, 0.423 [0.277, 0.583] on the defect-only n=83 intersection; McNemar p ≈ 6.0×10⁻⁸. Llama 3.2 Vision 11B is reported in Appendix A only owing to a documented multi-image input confound.