At the Threshold: Tracing Computational Hierarchy through Convergent Evidence in a Small Transformer

Language models now show a striking range of capabilities, from open-ended text generation to tasks that resemble reasoning, and behind nearly all of them sits the same architecture: the transformer. Much of the effort to understand these systems, however, has gone into measuring what they can do rather than into explaining how the underlying computation produces that behavior. Mechanistic interpretability asks the latter, tracing behavior to internal circuits and representations rather than treating the network as opaque. Progress here has come from two directions that rarely meet: reading attention patterns as circuits routing information and reading hidden states as representations accumulating meaning across depth. Whether these describe one computation or two separate, non-contradicting stories remains unclear. We investigate this using NanoLens, a small, fully inspectable character-level transformer. Using the NanoLens framework, we extract attention patterns and hidden-state trajectories from shared forward passes, building a taxonomy of six attention circuit types, quantitatively characterizing the resulting taxonomy with Shannon entropy, introducing the Behavioral Complexity Index (BCI), and relating these findings to measures of representational growth and stability across depth. Together, the attention and hidden-state analyses converge on a shared picture: a threshold where processing turns from local to abstract, a developing sequence-initial anchor point, and a broader shift toward specialization with depth. This convergence offers modest evidence that small character-level transformers remain a useful setting for generating testable, cross-checked mechanistic hypotheses. Code for all experiments is available at >>NanoLens Repository