cad-silent-failure-bench v1.1: benchmark, harness, and attempt corpus for "Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions"
Authors/Creators
- 1. Celabe
- 2. Hochschule Coburg, University of Applied Sciences
- 3. Independent
Description
Benchmark, harness, and full attempt corpus for the paper "Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions" (Rodrigues, Rodrigues, Reddy G, 2026). Suite v1.0 is the frozen version every number in the paper is computed from.
Contents: 13 natural-language mechanical specification tasks in three tiers with 122 machine-checkable requirements and published tolerance bands; 13 reference build123d solutions; the grader, kernel executor, property extractor, and single-shot / tool-use / multi-agent harnesses; every graded attempt (519 full-population attempts, the 468-attempt primary population, 156 legacy-contract ablation attempts (all four models), 60 pilot attempts) as JSON records with final code, measured properties, verdicts under the original and hardened oracle, completion claim, stated confidence, tokens, and context-divergence scores; multi-turn transcripts for the 312 fixed-protocol tool-use and ablation attempts; the blind expert scoring sheet; the analysis scripts that reproduce every statistic, table, and figure (including the tolerance-band sensitivity sweep); the pre-registered study plan; a static 3D viewer of every attempt against its reference; and the manuscript with its datasheet supplement.
A silent failure is a valid, renderable solid, delivered with an explicit completion claim, that fails at least one semantic check. Headline (four vendors, n = 156 per condition): single-shot agents were silently wrong on 14.1% of attempts; a tool-use loop that enforces measure-then-claim cut that to 3.2%; hardening the oracle flipped 18 of 519 verdicts from pass to fail and none in reverse, so these rates are lower bounds; scaling every tolerance band from x0.25 to x4 leaves the contrast intact.
v1.1 (2026-09-05). Claim-contract ablation extended from two models (n = 78) to all four (n = 156; new files data/ablation_legacy_sonnet.jsonl and data/ablation_legacy_qwen.jsonl with transcripts): silent failures 14/156 under the legacy contract vs 5/156 under the fixed contract (Fisher p = 0.056), claim precision 67.6% vs 96.0% (p = 4.5e-10), success 64.1% vs 77.6% (p = 0.013). Tolerance-band sensitivity table and script added. Paper PDFs refreshed to the revision-ready text. Files: cad-silent-failure-bench-v1.1.zip (code, tasks, data, docs, paper), cad-silent-failure-bench-v1.1-viewer.zip (viewer without models), cad-silent-failure-bench-v1.1-viewer-models-1.zip and -2.zip (the viewer STL models, split into two archives); unzip all four into one directory to reproduce the tagged tree v1.1. Details in ERRATA.md.
Licences: code MIT; tasks, reference solutions, records, transcripts, and viewer assets CC BY 4.0. Corrections are logged in ERRATA.md; later suite versions are separate tagged releases. Repository: https://github.com/rodriguescarson/cad-silent-failure-bench. Contact: carson@celabe.com.
Files
cad-silent-failure-bench-v1.1-viewer-models-1.zip
Additional details
Related works
- Is identical to
- Software: https://github.com/rodriguescarson/cad-silent-failure-bench/tree/v1.1 (URL)
- Is supplement to
- Software: https://github.com/rodriguescarson/cad-silent-failure-bench (URL)
Software
- Repository URL
- https://github.com/rodriguescarson/cad-silent-failure-bench
- Programming language
- Python
- Development Status
- Active