Physics-Informed Trajectory Constraints in Transformer Residual Streams

We investigate whether physics-informed trajectory constraints—soft penalties borrowed from
the PINN framework—can be imposed on a transformer language model’s residual stream during
training without degrading language modeling performance. We attach a lightweight scalar readout head to the residual stream and penalize trajectories that violate specified dynamical laws: monotonicity (the readout can only increase) and exponential decay (the readout fades
between events). Across 66 training runs spanning two datasets (TinyStories, WikiText-103),
two model scales (10.8M, 25M parameters), and four conditions (vanilla, multitask, monotone,
decay), we find that trajectory constraints cost 0–1.9% perplexity, with the cost shrinking
along both scale and training duration. Constrained models exhibit expanded representational
dimensionality (+20–37% PCA participation ratio), achieve target recoverability r 15 2 > 0.90
across constraint types, and—most strikingly—preserve features that unconstrained training
spontaneously develops and then degrades. Our results establish PINN-style trajectory constraints
as a viable, low-cost mechanism for imposing structural properties on transformer representations
at training time