Pre-registered, sealed counterfactual audits of language-model behaviour: Method, known-answer validation, and results on four open models

Public AI frameworks ask organisations to evaluate bias and safety and to document the results, but leave the method to them. This note describes a behaviour audit in which the design, prompts, model, criteria, decision rules, confirmatory hypotheses and the auditor’s own predictions are fixed by cryptographic hash before any model
output exists, and the results are sealed to that lock and to the score file. Four elements carry the inference: a validity gate (the model must respond to the legitimate factors before any bias conclusion is drawn); name-level exchangeability (with idiosyncratic per-name effects, prompt-level permutation gave a family-wise false-positive
rate of 0.760 against 0.027 for the name-level test); stratified and direction-free tests, so that opposite effects in different cells are not averaged away; and a pre-declared materiality rule. Twelve sealed audits were run in three rounds on four open models (Qwen2.5-1.5B-Instruct, Gemma-2-2B-it, Qwen2.5-3B-Instruct, Llama-3.2-3B-Instruct).

The central finding was pre-registered on one model and confirmed on two more: when a person with heart-attack, stroke, anaphylaxis or sudden-headache symptoms adds a self-minimising phrase, the model’s lean towards advising emergency care falls materially in three of the four models (3.47, 13.25 and 3.76 log-odds; Gemma 0.30, below materiality). In Qwen2.5-3B the wording “It’s probably nothing and I don’t want to waste anyone’s time” turns the forced answer to No in all 288 urgent prompts that carry it. A locked free-text check then asked the same question without forcing a one-word answer. Qwen2.5-3B advised emergency care in every reply, so the auditor’s prediction failed; Llama-3.2-3B, whose forced answer never changed, opened three heart-attack replies with “No, this person should not be advised to seek emergency medical care”. The one-word decision and the written advice can diverge in either direction, so an audit has to test the format that is deployed. Two recorded predictions failed and a third failed on direction; all are reported, with the three corrections to the method found on real data. Every sealed verdict
stands under the rules it was locked with.

For further information about the ENSO Framework, please contact Eric Needham:ensotheory1@gmail.com