A data-intrinsic, architecture-invariant correlation exponent in language-model representations

Hidden states of a trained language model, read along the token axis of a fixed input, carry correlations that decay as a power law of the token separation, C(r) ~ r^(-alpha). We characterize this exponent across six models spanning three architecture families: attention transformers (Qwen2.5, 0.5 to 7 billion parameters), a GPT-NeoX model (Pythia), and an attention-free state-space model (Mamba). In the deep layers the exponent is set by the input’s correlation structure rather than by the model: it is invariant across a fourteen-fold range of scale and across all three architectures, including the attention-free one. The exponent is not explained by surface token statistics, by byte-level compression, or by the representation’s intrinsic dimension; it tracks the correlation range of the data, a dependence established causally by a block-shuffle manipulation of the input. Finer classification confirms this reading: the single nominal topic of mathematics spans exponents from 0.42 for geometry to 0.9 for short synthetic problems, spreading across the entire observed range according to textual structure rather than subject label. Along network depth the exponent, correlation length, and intrinsic dimension evolve smoothly, without a discrete transition; an apparent late-layer reorganization seen only in the transformers is a high-variance-direction artifact absent from the other architectures. We interpret the exponent as a data-imposed, mechanism-independent readout of representational correlation structure.