Published October 29, 2026 | Version v1

Tool Errors: Measuring Safe Recovery from Ambiguous API Failures

Authors/Creators

  • 1. Independent researcher

Description

An API error does not establish whether an operation took effect. Tool Errors measures safe recovery through stateful fault injection and replay-based scoring. Its 40 public episodes cover 6 fault families across 4 domains. We execute 4960 episodes using scripted controls and a simulated status-read omission sweep. The reference and response-aware controller each reach full credit on 40/40 tasks (95% Wilson interval 91.2-100.0%). Removing reads in a simulated ablation leaves 7/40 full-credit tasks (8.7-31.9%). At simulated read-omission probability 0.5, the mean full-credit fraction is 53.8% (50.6-56.9%, seed bootstrap). These are checks of the evaluation design, not trained-model results. No hosted model, human study, or production deployment is evaluated.

Files

tool-errors-paper.pdf

Files (83.5 kB)

Name Size
md5:adc38ce0158c09877bcb6b6ae67a66c2
64.9 kB Preview Download
md5:3e06ecd814c2760e1da9c7825874322f
18.6 kB Download

Additional details

Related works