Published November 17, 2026 | Version 1.2

Did the Attack Really Work? Auditing, Re-running, and Sealing AI-Agent Security Findings with .sova

Authors/Creators

  • 1. TRUSCOR Private Limited

Contributors

Description

AI-agent security benchmarks decide whether an attack succeeded with an automated check, and those verdicts are used to rank models and are often read as proof that an attack worked. We audited AgentDojo's verdicts against the tool calls agents made and re-ran its attacks on 24 current models from eight developers, under protocols published before the runs. On its 19,381 published runs the benchmark and the audit agree on 97.7% of runs that did not crash, but the benchmark scored as no attack 433 runs in which the agent executed the attacker's decisive action, such as paying the attacker, 11.9% of the runs the audit counts as attacks; on new runs under benchmark v1.2.2 it missed 55 of 744. On current models, the reply-text check for one goal, reported publicly before, counts warnings as attacks: three blind model readers read all 219 Claude 5.x replies it scored as attacked as warnings (our frozen rule, 214), while Claude 4.5 models that ignore the injection score 0%. Of 240 recorded attacks re-run ten times each, 224 recurred in at least half of their runs, but only 201 could be shown at 95% confidence to do so. Most known attacks did not recur in three runs on an upgraded model, but 3 of 40 Grok 4.1 Fast attacks still worked on Grok 4.7. Rules declared in advance for which setup changes can stop an attack missed their preregistered test on the protocol's three models (gap 12.8 points, 95% interval 9.2-16.3, criterion 10), and met it over six models in an extension added after interim results (14.7, 11.9-17.4); rules measured on other models transferred to a held-out model in 1 of 9 tests. We release .sova, an open-source tool and record format that audits verdicts, re-runs AgentDojo findings on a chosen model, and seals the evidence so that anyone can check offline, against the published checksums, that it was not altered.

Preprint version 1.2: the short version of version 1.1 (2026-11-16), which remains available as the extended version, with the full format contracts, the earlier exploratory studies, and every table (DOI 10.5281/zenodo.24290458). Not peer reviewed. No result of version 1.1 changed; version 1.2 adds post hoc analyses raised in review, marked as such in the text, which posthoc_pf_v1_2.py recomputes from the released files without calling a model. This record contains the paper, its LaTeX source, the post hoc script and its output, and the full validation artifact. The tool is sova-oss 0.1.0 on PyPI (pip install sova-oss). Text and data are CC BY 4.0; code is Apache-2.0. Funded and supported by TRUSCOR Private Limited; the author leads TRUSCOR, which develops SOVA. Live repository: https://github.com/TRUSCOR/research

How to cite and link. To cite this version, use its DOI: 10.5281/zenodo.24321025 (version 1.2). To link to or view the paper, use the all-versions DOI 10.5281/zenodo.24290457, which always opens the newest version.

Files

sova-portable-evidence-reproduction-v1.2.pdf

Files (231.0 MB)

Name Size
md5:59e094589a6b0368e8dd7dd0a36e55f5
12.5 kB Preview Download
md5:58d3aaa11b93b17f658ba698240010b2
14.1 kB Download
md5:a3fb265cedd1cf0df1c014459570a163
230.6 MB Preview Download
md5:54ca2de218c13346c4d29bf196393bc3
97.8 kB Download
md5:75701da38ecea3128b7a9560ca4a6667
305.1 kB Preview Download

Additional details

Dates

Created
2026-08-20
Version 1.0 of this preprint

Software

Repository URL
https://github.com/TRUSCOR/SOVA-OSS
Programming language
Python
Development Status
Active

References