Testing Surprisal as a Compute-Allocation Signal: A Preregistered Long-Horizon Study of Causal Token Routing

Adaptive-computation systems are often motivated by the intuition that difficult or surprising tokens should receive additional processing. This preprint tests whether causal observed surprisal can serve as a reliable signal for allocating additional computation within a byte-level hybrid attention/Mamba-2 language model.

The evaluated system separates three mechanisms that are often conflated: a parameter-free ranker that identifies positions with high observed surprisal, a 705-parameter network that predicts the compute budget for the next chunk from the previous chunk’s surprisal profile, and a sparse contextual refiner that applies additional computation only at selected positions. The primary method is compared with a pure backbone, compute-matched fixed contextual refinement, fixed and dynamic token-MLP refinement, low-surprisal routing, and a content-independent position control.

Seven experimental arms were independently adapted for 50,000 steps from three paired warm starts. Complete evaluation over 4,999,936 development target bytes was performed at 5,000, 10,000, 25,000, and 50,000 steps, yielding 21 long-horizon training runs and 84 complete-development evaluations.

Although the primary candidate showed an advantage over the paired pure backbone at 25,000 steps, this advantage disappeared at the preregistered 50,000-step endpoint. None of the five required endpoint superiority comparisons passed. The learned budget policy also remained nearly constant and did not show the preregistered positive relationship between previous-chunk surprisal and next-chunk compute allocation.

Because both the quality and mechanism gates failed, the sealed final test was not opened. The reported results are therefore development evidence rather than a final-test performance claim. Under the evaluated architecture, scale, dataset, and optimization protocol, observed surprisal was not validated as a proxy for the marginal utility of additional computation.

This document is a preprint and has not undergone peer review.