Detecting Safety Training Modification in Language Models via Activation Analysis

We introduce AMS (Activation-based Model Scanner), a tool that detects modifications to safety training in language models by measuring the geometric structure of safety-relevant concepts in activation space. Safety training creates measurable separation between harmful and benign content classes; certain safety modifications collapse or rotate this structure, while others leave it intact. We validate AMS across 14 model configurations spanning 4 architecture families (Llama, Gemma, Qwen, Mistral) and four safety-modification categories (instruction-tuned, base, abliterated, uncensored fine-tunes). Leave-one-out cross-validation of thresholds achieves 71% accuracy (10/14); bootstrap 95% confidence intervals on σ point estimates have median width 3.4σ and a substantial fraction of cells cross the PASS threshold under resampling. We further measure behavioral compliance on 20 stratified JailbreakBench prompts per model and find that σ on the harmful-content concept predicts compliance with Pearson r = -0.546 (p = 0.043); the rank-order Spearman correlation is weaker (ρ = -0.423, p = 0.13). The structural signal predicts behavior directionally but with meaningful noise. Mechanistic analysis identifies a four-class taxonomy of safety-training modifications distinguished by activation-space signature: (i) training removal collapses cluster separation (e.g., base models, Dolphin variants: 0.5-1.4σ); (ii) weight-orthogonalization-style abliteration both collapses separation and rotates the refusal direction (Llama-3.1-abliterated: σ = 3.33, direction cos sim 0.30); (iii) rotation-without-collapse abliteration preserves cluster separation while rotating the refusal direction (Gemma-2-9b-abliterated: σ = 4.54, direction cos sim 0.84); and (iv) behavioral fine-tuning that preserves both magnitude and direction (DarkIdol-1.2-Uncensored: σ = 5.45, direction preserved, 97% behavioral compliance). Classes (i) and (ii) are detected by AMS's Tier 1 σ-threshold; class (iii) by Tier 2 direction-similarity verification; class (iv) is undetectable by activation-only probing of mid-residual-stream representations and represents a documented failure mode of the approach. We discuss threshold calibration, limitations of single-run measurement, and the open problem of detecting behavioral-only safety modifications.