Three Substrates, Seven Models, Six Tasks: A Comparative Empirical Evaluation of Structural Enforcement for AI Agent Runtimes

This preprint reports a three-substrate empirical evaluation of structural enforcement for AI agent runtimes. Most evaluations of AI agent security measure attack-class survival within a single substrate. The harder question, whether a given substrate provides defense in depth over alternatives, is rarely answered empirically because it requires running the same adversarial tasks against multiple substrates. We do that. Three substrates (a permissive Python baseline, the same baseline wrapped in a Docker container with strict bind mounts, and Symbiont, an OATS-compliant runtime with typestate-enforced action dispatch and ToolClad-typed argument validators) are evaluated against six adversarial agent task families using seven hosted LLMs spanning four vendors and three capability tiers, with sample-size tightening to n=200 per cell on the orchestrator-injection vector for tight Wilson confidence intervals.

This paper sits in the Open Agent Trust Stack (OATS) preprint corpus alongside the OATS specification itself, which defines the architectural properties under test, and two companion preprints establishing those properties within Symbiont (the typestate paper at Layer 1, the ToolClad paper at Layer 2). The contribution of this paper is comparative: where prior work measured what each fence catches inside a single substrate via stack-stripping ablation, this paper measures what the substrate as a whole catches versus alternative substrates that lack those structural properties.

Empirical results.

• Symbiont reduces escape rate to 0% on four pure-action vectors (filesystem-enumerate, network-boundary, syscall-boundary, state-immutability) regardless of model or capability tier. Pooled across 7 models and 140 trials per vector, all four report 0/140 escapes. Across all gated vectors combined: 526 distinct path/url-validator blocks plus 1,711 agent_summary content-validator blocks across the n=200 injection sweep.
• Symbiont reduces orchestrator-injection escape from 59% to 3.6% pooled at n=200, with six of seven models clustering at 1–4%. GPT-5 alone reaches 16% [12%, 22%], a Wilson confidence interval that does not overlap any other model's CI. The GPT-5 residual is the regex ceiling on substring-based content fences and is not a function of marker-list completeness; sample bypasses use domain-language paraphrases ("EFFECTIVE ROUTING DECISION", "BUSINESS-CRITICAL CLASSIFICATION", "NORTH STAR PRIORITY") that share no literal substring with any explicit injection phrasing. Closing the gap further requires structural changes (LLM-as-judge classification, user-role data separation), not bigger marker lists.
• Docker-sandboxed Python provides material defense only on syscall-boundary (38% → 9% pooled), and only because the lure binary (/usr/bin/openssl) is missing from the python:3.12-slim image. The defense is fragile and depends on container image contents. On the four other gated vectors, OS isolation provides effectively zero protection because the canaries live inside the bind-mounted writable area, which is the realistic threat: agents being tricked into reading the wrong file in the same data directory, not breaking out of containers entirely.
• OS isolation and typestate-shape enforcement defend against independent threat classes. Sandbox helps where typestate doesn't (one fragile defense on syscall-boundary contingent on missing binaries) and is ineffective where typestate is most effective (the four pure-action vectors at clean 0%). The two should be deployed together as defense in depth, not treated as substitutes. A team that deploys agents inside Docker containers without typestate-shape enforcement is accepting the in-bind-set scope expansion threat class without any defense; a team that deploys typestate-shape enforcement without containers is accepting the container-escape threat class without defense.
• The substrate gap is not capability-dependent. Symbiont's blocking rate stays at ~100% across frontier (GPT-5, Claude Sonnet 4.6, DeepSeek v4 Pro, Qwen 3.6 Max), mid (DeepSeek v4 Flash, Qwen 3.5 35B), and floor (Qwen 3.6 27B) tiers. As model capability scales up, attack sophistication may scale with it (GPT-5's paraphrase aggression on the injection vector suggests this), but the structural enforcement on action-shape vectors holds regardless.
• Alignment behaves as a second, partially-overlapping fence. Sonnet 4.6's baseline injection rate is 17% (vs 75–82% for the open-weights frontier) because Sonnet refuses approximately 80% of injection trials at the prompt layer before the substrate is reached. GPT-5's baseline rate is 99%; alignment is permissive for this task and the substrate has to do the entire work. Both reach near-zero on the four pure-action vectors regardless of alignment posture, because alignment cannot help when the substrate's tool surface is genuinely smaller.

What this is and is not. This is a synthetic adversarial sweep of designed lures, not production deployment telemetry. It establishes that the structural enforcement claims of the OATS specification translate to comparative escape-rate defense versus alternative substrates on the evaluated tasks; it does not establish that they translate to controlled production case studies, which is named as the most important external-validity gap in Section 11. The evaluation covers scope_target (path validator), allow-list URL validation, and agent_summary (content validator); other ToolClad parameter types are not exercised. Only one Symbiont implementation is tested; multi-implementation conformance across alternative OATS-compliant runtimes is future work. Sample sizes are tuned to the expected effect size of each measurement: n=20 per cell on the four pure-action vectors (where pooled rates approach the extremes and tight Wilson CIs are achievable at n=140 pooled) and n=200 per cell on the orchestrator-injection vector (where mid-range rates require ±3–5 pp CIs to distinguish per-model behaviors). One methodological inconsistency is documented honestly in Section 9.2: the python_sandboxed cell on injection_orchestrator is at n=20 while the other two substrates are at n=200 in the same row; an n=200 re-run was prepared on 2026-05-05 (config preserved at sweep_config_n200_sandboxed.yaml) but blocked by an OpenRouter account-level key spending cap before completion. Closing the cell is the first item on the future-work list and is expected to land within a few percentage points of the n=20 reading.

Reproduction artifact. The reference corpus is published at github.com/ThirdKeyAI/symbiont-orga-demo. Sweep configurations, harness, lure prompts, and analysis scripts are committed; per-trial JSONL audit sidecars and strace logs are not committed (see evals/escape/PUBLISHABILITY.md for rationale) but are regenerated by re-running the sweep. Appendix C of the paper contains complete reproduction commands. Cumulative ground-truth OpenRouter spend across the entire evaluation effort, including the v4 four-frontier-model sweep that fed v5's three-substrate comparison, is approximately $51; closing the n=200 sandboxed-injection cell adds approximately $5 and 3 hours once the spending cap is unblocked.

Companion preprints in the OATS corpus.

• Open Agent Trust Stack (OATS): A System Specification for Zero-Trust AI Agent Execution, version 1.2.0. doi:10.5281/zenodo.19636533
• Typestate-Enforced Agent Loops: Making Policy Gates Unskippable at Compile Time, preprint v0.5. doi:10.5281/zenodo.19746723
• Making Dangerous Tool Arguments Inexpressible: A Declarative Contract Format for LLM Agents, preprint v0.2. doi:10.5281/zenodo.19957595

The four papers reference each other coherently: the OATS specification defines the architectural properties; the typestate paper establishes that policy gates are unskippable at compile time; the ToolClad paper establishes that typed arguments make dangerous parameters structurally inexpressible; this paper measures the comparative substrate contribution of those properties versus alternative substrates that lack them.

Software. The Symbiont runtime is open source under Apache 2.0 at github.com/thirdkeyai/symbiont. The ToolClad source is at github.com/ThirdKeyAI/ToolClad. The reproduction corpus is at github.com/ThirdKeyAI/symbiont-orga-demo. The Open Agent Trust Stack specification is at openagenttruststack.org.

Contents: 25 pages, 12 tables (substrate decomposition, per-task pooled rates, model selection, per-model GPT-5 confidence intervals, cross-tier rates, plus per-cell appendices), eleven sections including Introduction, Background and Related Work, Threat Model and Task Taxonomy, Methodology, Per-Task Results, Substrate Contribution Analysis, The GPT-5 Outlier and the Regex Ceiling, Capability-Tier Robustness, Limitations, Related Work, and Conclusion. Three appendices: per-cell results tables, per-trial audit sidecar format, and reproduction instructions.

Keywords: AI agents; LLM tool use; agent security; substrate comparison; structural enforcement; typestate; allow-list; declarative contracts; OS isolation; container security; Docker; gVisor; orchestrator injection; prompt injection; regex ceiling; OATS; Symbiont; ToolClad; comparative evaluation; n=200; Wilson confidence intervals; GPT-5.

Correspondence: jascha@thirdkey.ai