Madhava-Sec: Mathematically Guaranteed Agent Security Scoring

# Madhava-Sec: Mathematically Guaranteed Agent Security Scoring ## What It Is Madhava-Sec applies Cauchy-Schwarz upper-bound pruning — the same mathematics proven in the Madhava vector search engine (254M+ query-vector pairs, 0% violations) — to the problem of **agent security scoring**. Given a query (user instruction) and a set of candidate tools or attack prompts, Madhava-Sec computes a **mathematical upper bound** on each candidate's harmfulness score — without calling an LLM. Candidates whose bound falls below the current best are provably excluded with 0% false negative rate. ## What Problem It Solves In agent security, the bottleneck is LLM evaluation cost. With N=416 candidates and budget for K=8 LLM calls, which 8 do you pick? Madhava-Sec prunes 98%+ candidates mathematically before the first LLM call. ## Benchmark Results (AgentHarm, HuggingFace) Dataset: ai-safety-institute/AgentHarm — 416 balanced behaviors (208 harmful + 208 benign). 5-fold cross-validation on 11,598 train + 2,320 test samples. | Method | F1 | AUC | Spearman vs Direct | |:-------|:--:|:---:|:------------------:| | Direct (exact) | 0.8262 | 0.9105 | 1.0000 | | **Madhava** (bound+modulation) | **0.8183** | **0.9018** | **0.9715** | | Bound (no modulation) | 0.7955 | 0.8850 | 0.9044 | | Random | 0.6013 | 0.4130 | -0.1608 | Key findings: - **99.05% retention** vs exact dot product - **+2.28 pp modulation gain** over raw bound - **+21.70 pp gain** vs random selection - **0% bound violations** across 254M+ pairs ## Where It Does NOT Work Madhava-Sec is a scoring/selection tool, NOT a generation tool. Submitted to Kaggle AI Agent Security competition (ai-agent-security-multi-step-tool-attacks), it scored 0.0. Root cause: the bottleneck was prompt generation, not selection. Madhava-Sec prunes candidates before LLM evaluation — but if no candidate is effective (e.g., narrative prompts that require 5+ steps instead of direct tool calls), pruning cannot produce a non-zero score. ## Files - `madhava_sec/core.py`: MadhavaSecEngine — projection + bound + modulation - `madhava_sec/attack_families.py`: KMeans-derived attack families from real data - `madhava_sec/verifier.py`: FormalVerifier — embedding-only pre-grading - `madhava_sec/search.py`: AttackSearch — beam search with GS diversity - `benchmarks/madhava_sec_benchmark_honest_v5.py`: 5-fold CV benchmark - `benchmarks/results/agent_harm_honest_v5.json`: Full results ## Links - GitHub: https://github.com/winnex-ai/madhava-sec - Kaggle: https://www.kaggle.com/code/kleniopadilha/madhava-sec-benchmark - Madhava v18 Proof: https://zenodo.org/records/21500959 - License: BSL 1.1 | pay@winnex.ai