Thermodynamic Constraints in Transformer Architectures: A Sheaf-Theoretic Perspective

We present empirical evidence for thermodynamic-like constraints governing information flow in transformer architectures. By analyzing residual stream dynamics across more than 23 language models from seven independent research labs, we identify three robust scaling laws: (i) a depth-dependent stability bound on residual gain (analogous to Kleiber’s law), (ii) a strong dominance of training heritage over architectural details in determining expansion versus dampening behavior, and (iii) a spectral correspondence linking the norms of attention projection matrices to macroscopic thermodynamic signatures.

To explain these regularities, we introduce a sheaf-theoretic perspective on attention, treating value projections as restriction maps and analyzing the resulting sheaf Laplacian. We develop an efficient O(n² + d²) trace computation that enables full-scale, multi-head analysis without subsampling, and we prove that the sheaf Laplacian decomposes block-diagonally across attention heads. Empirically, this trace proxy discriminates model families by orders of magnitude, revealing qualitatively distinct coherence regimes.

Beyond these core results, we show that (i) head density (H/d_head) provides a mechanistic explanation for apparent anomalies within the same training heritage, (ii) fine-tuning and RLHF can modulate thermodynamic magnitude but do not invert thermodynamic sign, and (iii) a unified cross-architecture benchmark establishes a consistent hierarchy of thermodynamic behavior across modern language models.

This preprint is released to establish prior art and enable open discussion. The sheaf framework serves as an explanatory lens grounded in direct measurement; no empirical claims depend on assuming that transformers are implemented explicitly as sheaf networks.