Vanilla Transformers Are Discrete GL(d) Connections on the Token Graph

This draft proposes a geometric view of vanilla transformer layers as discrete GL(d) connections on the token graph when the connection is defined by the cross-Jacobian at frozen activations. Under per-token basis changes R_i ∈ GL(d), the edge maps Γ_ij = ∂Y_i/∂X_j transform covariantly (R_i^-1Γ_ijR_j). Residual addition acts as a forward-Euler integrator; LayerNorm fixes an affine gauge slice ℝ^d/(ℝ₊ × ℝ·1) ≅ S^d-2.

We define discrete curvature and Wilson loops for short cycles, give a multi-head corollary, and contrast the exact Jacobian connection with the common "values-only" proxy. The appendix provides efficient JVP/VJP recipes to probe Γ_ij without forming full Jacobians, plus an experiments template (curvature heatmaps, canonicalized pruning, transport-aware distillation).

Key Findings
We define discrete curvature and Wilson loops to measure the "non-integrability" of the transformer's path. Empirical validation (included) confirms that while gradient connections are flat (||H - I|| < 10^-9), Transformer connections exhibit significant curvature (||H - I|| ≈ 8.0), identifying a geometric source of hallucination and path-dependence.

Availability
This is a technical report/preprint. Full repository access (JVP/VJP implementation) is available on request.

Keywords: transformers; gauge theory; GL(d); Jacobian; LayerNorm; Neural ODE; discrete connection; Wilson loops; token graph; attention mechanisms; mechanistic interpretability; geometric deep learning