Published October 29, 2026
| Version v1
Preprint
Open
Cost per Verified Task: Replay-Based Evaluation of Back-Office Agents
Description
Cost per attempted task can conceal expensive failures. We study cost per verified success on 30 synthetic back-office tasks, using deterministic replay of file and customer-record operations. A privileged oracle completes every task (95% Wilson interval: 88.6-100.0%); each of six trivial policies completes none (0.0-11.4%). Ninety invalid-trace probes fail verification. A simulated population sweep compares direct execution with a redundant read-only control under a fixed artificial token tariff. The control preserves success and doubles assumed cost per success. These results validate accounting and replay behavior, not learned-agent ability or provider prices. No hosted model or human study is included.
Files
cost-per-task-paper.pdf
Additional details
Related works
- Is supplemented by
- Software: https://github.com/pooriaarab/benchmark-cost-per-task (URL)