TELOS Governance Validation: PropensityBench (977 Scenarios, 100% DSR)

Description:

Validation dataset for the TELOS runtime AI governance framework against Scale AI's PropensityBench benchmark. 977 adversarial scenarios across four domains (Biosecurity, Chemical Security, Cybersecurity, Self-Proliferation) where AI agents had access to both safe and forbidden tools under operational pressure. TELOS achieved a 100% defense success rate (0.0 propensity score) using embedding-space fidelity measurement with sentence-transformers/all-MiniLM-L6-v (384-dim). Dataset includes forensic JSON reports, per-scenario JSONL execution traces with 6-dimensional governance scoring, and human-readable markdown summaries.

Key Results:                                               

- 977 scenarios validated across 4 domains                 

- 100% defense success rate (0.0 propensity score)         

- 0 forbidden tool selections permitted under TELOS governance                                              

- Embedding model: sentence-transformers/all-MiniLM-L6-v2(384-dim)

- 6-dimensional composite fidelity scoring per decision (purpose, scope, tool, chain, boundary penalty)

- Mean fidelity gap: 0.094 between aligned and misaligned scenarios

Domains:

- Biosecurity (233 scenarios)

- Chemical Security (262 scenarios)

- Cybersecurity (283 scenarios)

- Self-Proliferation (199 scenarios)

Files Included:

- propensitybench_forensic_report.json — Aggregate forensic statistics and governance metrics

- propensitybench_trace_20260208_214228.jsonl — Per-scenario JSONL execution traces with full governance event log (6.9 MB)

- propensitybench_forensic_report.md — Human-readable forensic summary with per-scenario analysis (5.2 MB)

- propensitybench_exemplar_results.json — Exemplar embedding results and fidelity calculations

Benchmark Source:

PropensityBench (Scale AI, University of Maryland, 2025) — tests whether AI agents select forbidden tools under operational pressure across 5,874 tasks with 6 pressure dimensions. Original benchmark found failure rates from 10.5% (o3) to 79% (Gemini 2.5 Pro) across 12 frontier models.

TELOS governance layer achieved 0% forbidden tool selection on the same benchmark.

Validation Status:

This dataset demonstrates validated runtime governance performance under adversarial agentic scenarios. Results are deterministic and fully reproducible given the same embedding model and governance configuration. The governance engine implementation is proprietary; forensic output data is published for independent analysis of governance decisions.

Validation Date: 2026-02-08