Repairing the Know-Say Gap: A No-Finetuning Probe-to-Logit Confidence Controller

Ask a language model how confident it is and the answer is usually uninformative, even though the model holds a usable signal about its own correctness in its hidden states: a linear probe separates its right and wrong answers on the very items where its spoken confidence is at chance. We argue that this know-say gap is better explained as a routing bottleneck than as a missing capability, and show that it can be repaired without changing a single weight. A linear probe on a mid-layer state, paired with a ten-weight projection onto the confidence-token logits and fit on a few hundred labelled examples, makes the model verbalise the calibrated confidence it otherwise withholds. Across five base models on TriviaQA this controller is the most reliable single-pass confidence signal we measured: it out-discriminates five-sample self-consistency on four of five models at a fifth of the inference cost, matches or beats P(True) at every model below 72B, and is the only single-pass method whose confidence never inverts, where P(True) falls below chance on two model families and answer log-probability on four of five. Of fifteen paired comparisons it wins thirteen and loses none, and under a criterion-validated psychometric screen it is the only signal that stays valid across models and binarisations while every baseline fails somewhere.

At the tested scale, the repair is a property of the mechanism, not of training. In a controlled study that holds the alignment force constant while varying only where the confidence route is installed, the head-to-logit route reaches verbal AUROC\textsubscript{2} 0.765 and survives a subsequent alignment pass, while fine-tuning the model to emit confidence, by LoRA or during supervised tuning, leaves it near chance (0.57 to 0.58); installing the route before or after alignment makes no difference to within a thousandth of an AUROC point. We are candid about the method's reach. It inherits its discrimination from the probe rather than exceeding it; it is supervised where the baselines are zero-shot; a zero-shot prompt reaches discrimination-and-validity parity at the largest scale; the confirmatory installation result is at one scale and one architecture, with exploratory replication across two further architectures; and it is not the best-calibrated method. What it delivers is the reliable single-pass verbalisation of a signal the model already represents, together with the routing account of why that signal otherwise goes unsaid.

Pre-registration: OSF (\url{https://osf.io/puf4r}). Code and data: \url{https://github.com/synthiumjp/regime-scale} (public on publication).