Predicting How Transformers Attend, Part III: From Attention to Residual Computation — Separating Transport, Writing, and Commitment

Third paper in the “Predicting How Transformers Attend” series (Marín 2026), following Part I (Zenodo doi:10.5281/zenodo.19826343), which introduced the Thermodynamic Attention Framework and the closed-form predictor for the attention-decay exponent γ, and Part II (doi:10.5281/zenodo.19960573), the six-axis decomposition. Parts I and II characterised the distance axis of attention; this paper opens the orthogonal depth axis.

Its starting observation: the single interpretability question “is the answer here yet?” silently conflates four distinct statements about the residual stream — that information is present, linearly decodable, written, and used. The paper shows these are separate, independently measurable observables, and decomposes residual computation into three: transport (the mean input–output Jacobian; the Jacobian lens), net injection (direct logit attribution, DLA), and commitment (the depth at which the running decision locks). The DLA decomposition is exact: with the final normalisation frozen the output logit equals the sum of per-layer contributions (validated to 7.6×10⁻⁶, norm-aware for LayerNorm and RMSNorm).

Empirically the paper reports the only ladder-complete Jacobian-lens readability suite across a Pythia scale ladder (70M–2.8B) with cross-family anchors (Llama-2/3, Qwen2.5/3, GPT-2). The readability knee is a family property (Pythia 0.54–0.75 of depth, Llama 0.81–0.84) and the Jacobian-lens advantage band is strikingly heterogeneous (absent at d_head = 256; up to 68× at layer 1 of Qwen2.5-3B).

The paper's pivot is a structural theorem: the advantage band is not a “workspace” of stored content but a property of the transport operator itself (J_ℓ = ∏_k≥ℓ(I + w′_k)), which forces a strong anticorrelation (Spearman −0.80 for net injection, pythia-410m; replicated on pythia-2.8b and Qwen3-1.7b) between Jacobian-lens advantage and net injection. A generic denoised mean operator saturates as an estimator at N ≈ 20 prompts and already beats the per-prompt oracle, so the lens advantage measures generic linear transport, not prompt-specific content. This sharpens — and does not contradict — the causal “global workspace” result of Lindsey et al. (Anthropic, 2026): a lens advantage is a hypothesis generator, not causal evidence. Across families the observables generalise but their chronology does not: a single number, Δ_prep = τ_write − τ_decode, classifies families cleanly (Type A ≥ 0.5 vs Type B ≤ 0.3, no overlap).

Honest revisions, each by a pre-registered discriminant design and reported in the same register as surviving results: three hyperparameter predictors of the knee are refuted (the L_crit formula — discriminant pythia-1b, predicted 0.94, observed 0.688; a universal constant ≈0.69, killed by the full ladder; the RoPE fraction, inconclusive by a frozen control rule). L_crit is not discarded — it governs a distinct, causally-validated attention-regime transition, decoupled from the knee. τ_commit is architectural, not a deep law. The relation of the chronology to the Part I exponent γ is left inconclusive (two γ-measurement protocols disagree and the direction of any co-ordination flips between them). All experiments are pre-registered with frozen decision rules; the load-bearing algebraic identities are machine-verified in Lean 4/Mathlib in a companion formalization paper (a separate release). English (14 pages) and Spanish (15 pages) editions are included. Prepared with the assistance of an AI system (Claude, Anthropic); the research and all claims are the author's responsibility.