Published October 12, 2026
| Version 0.7.0
Software
Open
CACE-Bench: A Synthetic Agentic Evaluation Benchmark for LLM Credit-Pipeline Agents
Description
Compliance-Aware Credit-agent Evaluation (CACE-Bench): a synthetic, reproducible benchmark and generator for evaluating LLM-agent credit pipelines, and the self-evolution of such pipelines, under auditability constraints. No production credit data is used; the only real records are the 1,000 applications of the public, anonymised Statlog German Credit data (UCI, CC BY 4.0), shipped with a SHA-256 check.
Version 0.7.0 runs the regulatory-shift track on those applicants (evolve_real.py; evolve.py unchanged) with a 700/300 split of applicants per seed, so no evaluated applicant is in the gate's pool. Over 90 runs the gated dual loop admitted 171 of 6874 candidate changes with 0 harmful admissions; the same loops without the gate admitted 228 harmful changes and ended with a missed-flag rate above 10 percent in 29 of 90 runs. The dual loop reached the oracle harness in 80 of 80 runs outside the scope-high cell and in 0 of 10 within it, where the fixed tolerance blocks the correct change (9 of 10 at epsilon = 0.01); there the residual false positives fall unevenly, by +2.3 percentage points for women and +4.2 for applicants under 25 beyond the oracle's own gap. Version 0.7.0 also adds a language-model proposer (proposer_llm.py) with a response cache keyed by SHA-256 for exact replay. In a pilot of 15 runs (mid severity, 5 seeds; Claude Haiku 4.5 as proposer, model version and temperature not pinned, reproducible from the released cache) both loops ended at the correct harness, but without the gate 13 admissions were harmful and missed flags during adaptation reached 22.3 percent in the scope family, against 0 and 2.0 percent with the gate.
Version 0.6.0 added the regulatory-shift track (evolve.py): a supervisory re-interpretation at 70 percent of a stream of screening alerts, three families of re-interpretation (threshold, scope, structural) at three severities, a dual-loop engine that may change only the runtime harness, and an admission gate whose every decision is written to a hash-chained log before deployment. Ten arms per cell, including no gate, gate on pre-shift labels, oracle, manual update and compute-matched local-only; 10 seeds per cell. Over 90 runs the gated dual loop admitted 144 of 7449 candidate changes with 0 admissions that increased error on a held-out half of the retrospective pool; the same loops without the gate admitted 309 such changes and ended with a missed-flag rate above 10 percent in 49 of 90 runs. The proposer in this track is a seeded search over a stated edit space, not a language model.
The reference run (cace_bench.py, seed 0, N = 23,000, registry 2026-08-03) is unchanged and now reports the missed-flag rate beside the false-positive rate and is replicated on seeds 0-4: silent-decision rate 55.11 to 6.53 percent; compliance false-positive rate 22.51 to 4.96 percent; missed-flag rate 15.30 to 3.47 percent; hallucination 2.55 to 0.62 percent; provenance completeness 93.40 to 99.33 percent. The recovery arm is a documented parametric model.
Scope. All figures characterise reference components on a synthetic distribution and are illustrative of the method, not of production performance. Provider-coverage figures are working assumptions. The synthetic tracks omit protected attributes; the German Credit track reports error gaps by sex and age on one small public data set, a measurement rather than a fairness audit, and its screen is a referral policy, not a risk model. Every number in the accompanying paper is generated from the result files in this archive.
Files
rav11l/cace-bench-v0.7.0.zip
Additional details
Related works
- Is supplement to
- Software: https://github.com/rav11l/cace-bench/tree/v0.7.0 (URL)
Software
- Repository URL
- https://github.com/rav11l/cace-bench