Published September 5, 2026 | Version 1.1

cad-silent-failure-bench v1.1: benchmark, harness, and attempt corpus for "Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions"

  • 1. Celabe
  • 2. Hochschule Coburg, University of Applied Sciences
  • 3. Independent

Description

Benchmark, harness, and full attempt corpus for the paper "Done Is Not Correct: Measuring Silent Failures and Self-Verification Calibration When LLM Agents Take CAD Actions" (Rodrigues, Rodrigues, Reddy G, 2026). Suite v1.0 is the frozen version every number in the paper is computed from.

 

Contents: 13 natural-language mechanical specification tasks in three tiers with 122 machine-checkable requirements and published tolerance bands; 13 reference build123d solutions; the grader, kernel executor, property extractor, and single-shot / tool-use / multi-agent harnesses; every graded attempt (519 full-population attempts, the 468-attempt primary population, 156 legacy-contract ablation attempts (all four models), 60 pilot attempts) as JSON records with final code, measured properties, verdicts under the original and hardened oracle, completion claim, stated confidence, tokens, and context-divergence scores; multi-turn transcripts for the 312 fixed-protocol tool-use and ablation attempts; the blind expert scoring sheet; the analysis scripts that reproduce every statistic, table, and figure (including the tolerance-band sensitivity sweep); the pre-registered study plan; a static 3D viewer of every attempt against its reference; and the manuscript with its datasheet supplement.

 

A silent failure is a valid, renderable solid, delivered with an explicit completion claim, that fails at least one semantic check. Headline (four vendors, n = 156 per condition): single-shot agents were silently wrong on 14.1% of attempts; a tool-use loop that enforces measure-then-claim cut that to 3.2%; hardening the oracle flipped 18 of 519 verdicts from pass to fail and none in reverse, so these rates are lower bounds; scaling every tolerance band from x0.25 to x4 leaves the contrast intact.

 

v1.1 (2026-09-05). Claim-contract ablation extended from two models (n = 78) to all four (n = 156; new files data/ablation_legacy_sonnet.jsonl and data/ablation_legacy_qwen.jsonl with transcripts): silent failures 14/156 under the legacy contract vs 5/156 under the fixed contract (Fisher p = 0.056), claim precision 67.6% vs 96.0% (p = 4.5e-10), success 64.1% vs 77.6% (p = 0.013). Tolerance-band sensitivity table and script added. Paper PDFs refreshed to the revision-ready text. Files: cad-silent-failure-bench-v1.1.zip (code, tasks, data, docs, paper), cad-silent-failure-bench-v1.1-viewer.zip (viewer without models), cad-silent-failure-bench-v1.1-viewer-models-1.zip and -2.zip (the viewer STL models, split into two archives); unzip all four into one directory to reproduce the tagged tree v1.1. Details in ERRATA.md.

Licences: code MIT; tasks, reference solutions, records, transcripts, and viewer assets CC BY 4.0. Corrections are logged in ERRATA.md; later suite versions are separate tagged releases. Repository: https://github.com/rodriguescarson/cad-silent-failure-bench. Contact: carson@celabe.com.

Files

cad-silent-failure-bench-v1.1-viewer-models-1.zip

Files (11.5 MB)

Name Size
md5:f9d794b4c0a511924e4b9b702a4b8cb3
5.3 MB Preview Download
md5:2ceafad4f4163924cba7bbabed8cf445
5.3 MB Preview Download
md5:69baac24f8cf966991ea506f275188ea
100.0 kB Preview Download
md5:f9144a1799819779a07daba92797049f
819.9 kB Preview Download

Additional details

Software

Repository URL
https://github.com/rodriguescarson/cad-silent-failure-bench
Programming language
Python
Development Status
Active