Anytime-Valid Certification of Intervention Effects in Mechanistic Interpretability

The repository contains the code and data relevant to the article. All details are provided in the article. 

Abstract: Mechanistic interpretability often evaluates internal units with ablation, patching, and steering scores, but these are point estimates drawn from large, correlated candidate sets and are often monitored repeatedly. We formulate validation as a pre-specified test of two intervention effects: necessity under ablation and sufficiency under installation. Each contrast is mapped through a frozen bounded encoding and tested with a betting supermartingale, yielding an e-value valid under optional stopping; their minimum tests the conjunction, and e-BH controls the false-discovery rate over a declared family under arbitrary dependence. The certificate is therefore relative to the writer, token scope, input population, score, and candidate family, and does not identify a representation-independent mechanism. On GPT-2 indirect-object identification, FACE certifies four of forty-eight screened features. Sequence-wide writing increases sufficiency by up to fivefold relative to clean-value patching, and a one-thousand-candidate position-aligned analysis retains three. At a pilot-defined margin in correct-name probability, fixed before the probability-scale analysis, a secondary three-hundred-round replay yields no threshold crossing, whereas a fresh six-hundred-round run with the analysis locked before data generation certifies feature 3983 at round 394 (design estimate 391). On held-out prompts, the certified set shifts the model toward the withheld name while a pilot-activation-matched uncertified control shifts it away; it does not separate from the highest-ranked near misses. Decoded group e-values are invariant to coupled changes of basis within the tested span, unlike coordinate-level e-values. The S-inhibition head set certifies, while primary name-mover heads are declined under single-head ablation, consistent with previously reported self-repair.