Statistical Mechanics of Large Language Models: Free Energy, Order Parameters, and Collective Behavior

This paper develops a statistical-mechanical framework for understanding the collective behavior of large language models (LLMs). Building on companion work in topology, information geometry, and many-body physics, the study interprets internal representations as configurations in an empirical ensemble and analyzes their macroscopic organization through free energy, order parameters, entropy, and correlation structure.

The paper introduces effective free-energy functionals for internal representations, clarifies how modular specialization and rank collapse emerge as phase-like transitions, and defines order parameters that distinguish weakly and strongly specialized regimes. The curvature of empirical scaling laws is interpreted through free-energy saturation, while the geometry of the effective energy landscape is analyzed via its Hessian spectrum, revealing flat directions and reduced effective dimensionality.

The appendix provides a unified treatment of entropy, Bayesian evidence, partition functions, and singular learning theory, showing how Fisher degeneracy, spectral collapse, and Watanabe’s asymptotic results naturally align with statistical-mechanical principles. Together, these results offer a coherent theoretical framework for interpreting the structural organization of LLMs and for guiding future research on scaling, architecture, and representation geometry.