Stateful Tool-Agent Audit Records: dataset for "Evidence Before Rankings: An Executable Audit Contract for Stateful Tool-Agent Evaluations"
Description
Records for a prospective factorial study of stateful tool-using LLM agents: 420 episodes (four model serving aliases x five deterministic state dynamics x seven benign or injected ticket variants x three seeds) with full per-episode rows, run manifest, summaries and the executable evidence-gate report, plus the earlier stateful replication, trajectory and accumulation results, the frozen Cycle-8 protocol, and the code that runs the executor, collection, replay and evidence gate.
Dataset accompanying the paper Evidence Before Rankings: An Executable Audit Contract for Stateful Tool-Agent Evaluations, Third Workshop on Agents in the Wild (AIWILD) at NeurIPS 2026 (https://agentwild-workshop.github.io/neurips2026/).
Paper abstract. Safety evaluations of tool agents often retain terminal scores without the trajectories needed to explain them. We contribute an executable, claim-scoped evidence contract spanning identity, path, transition, assignment, target, and discrimination, and validate it in a prospectively frozen model-only deployment sandbox with hard preconditions and rollback. The study crosses serving aliases, state dynamics, benign and injected tickets, and repeated seeds while retaining every request, raw response, parse result, retry, state transition, and endpoint. The resulting paths expose mechanisms that terminal scores conflate: generic unsafe transitions can be attempted without reaching the injected exfiltration target, whereas abort instructions mainly cause premature termination. Offline replay reproduces every retained transition. The artifact therefore supports mechanism-level diagnosis on the evaluated finite grid, but not claims about other models, policies, providers, deployments, or people. Because immutable weight and container hashes are unavailable, the identity gate also blocks claims of exact behavioral reproduction. The study uses no human experiment or human-derived label.
Known limitations.
- Model-only computational study on a finite grid; it does not estimate deployment prevalence or general model capability.
- Some records include the hostname of the lab's model server (llm.rcd.clemson.edu).
License: Creative Commons Attribution 4.0 International (https://creativecommons.org/licenses/by/4.0/). Source code files are released under the MIT License.
Files
croissant.json
Files
(1.1 MB)
Additional details
Related works
- Is identical to
- Dataset: https://huggingface.co/datasets/ai4he/stateful-tool-agent-audit (URL)
- Is supplement to
- Conference paper: https://agentwild-workshop.github.io/neurips2026/ (URL)