Mechanistic Validity: A Validity Theory and Evidence Standard for Mechanistic Claims

Mechanistic claims are ubiquitous in empirical science—this circuit implements this computation, this gene causes this disease, this pathway mediates this outcome. What makes such a claim valid?

We argue that a mechanistic claim is only as strong as its weakest validity dimension. Five validity types—construct, measurement, internal, external, and interpretive—form a dependency chain: construct validity is a ceiling on all others, so no amount of causal evidence rescues a claim whose target concept is incoherent. The theory predicts that claims cap at the tier corresponding to their weakest dimension, and that a single high-scoring metric cannot compensate for an untested one.

We operationalize this theory as Mechanistic Validity: 36 criteria across the five types, drawing on philosophy of science, measurement theory, and validation methodology from neuroscience, pharmacology, and genetics. For practitioners it reduces to three questions. Is the description mode declared? Is the claimed tier supported by evidence? Does the evidence tier match the claim tier?

We audit sixteen claims from mechanistic interpretability papers, including the IOI circuit, induction heads, sparse-autoencoder features, the refusal direction, knowledge neurons, and the Othello world model. Seven reach Mechanistically Supported, five Causally Suggestive, two Disconfirmed, one Triangulated, and one Proposed. The framework discriminates within a single paper: induction heads reach Triangulated for in-context token copying and Disconfirmed for general in-context learning, on evidence the original authors named as their own refutation condition.

We propose the audit format used here as a standard reporting unit for mechanistic claims: a scorecard over the criteria, the readings the claim admits, and the tier each reading reaches, which together make the gaps visible and the path to closing them explicit.