Probing Algebraic Structure in Neural Networks via Symmetric Cycle Closure

## Abstract

Mechanistic interpretability of grokking [Power et al. 2022] often relies on label accuracy or geometric proxies to argue that a network has learned algebraic structure. We argue these signals are not enough and introduce **Symmetric Cycle Closure Count (Sym-CCC)**, a functional probe based on group axioms. It tests bi-directional cancellation: a network respects group $G$ if both $\text{Net}(\text{Net}(a,b), b^{-1}) = a$ and $\text{Net}(a^{-1}, \text{Net}(a,b)) = b$ hold over $G \times G$. We derive the random baseline analytically ($1/|G|^2$, Lemma 3.1) and confirm it on untrained networks.

We test this probe on two non-abelian groups of order 24 — the dihedral group $D_{12}$ and the symmetric group $S_4$ — across multiple seeds, and for $S_4$, across independently-randomized train/val splits with a complete, pre-registered $n=5$ seed set. Four findings:

1. **Weight decay separates algebraic generalization from algebraic overfitting.** On $D_{12}$ (5 seeds), removing weight decay yields 100% train accuracy with Train Sym-CCC = 53.9% ± 3.1% but Val Sym-CCC = 5.7% ± 1.8%. With weight decay (0.1), both rise to 79–92%. On $S_4$ (5 seeds, closed-domain evaluation with a fixed, task-derived denominator), the separation is sharpest: **exact-zero, zero-variance Val Closed Strict Sym-CCC** without weight decay across all 5 seeds, versus 54.7% ± 31.2% with weight decay, at a pre-registered 4/5 grokking rate.

2. **Coordinate alignment speeds up but does not decide structure learning.** On $D_{12}$, random index permutation reaches Val Sym-CCC = 72.6% ± 9.2% versus 79.4% ± 10.4% under contiguous indexing.

3. **A late-phase Float32 dip is linked to precision noise, not Softmax overflow**, on $D_{12}$, supported by two direct interventions (AdamW $\epsilon$, Float64). A matching dip observed on $S_4$ is reported as an unconfirmed, observational parallel.

4. **The core separation replicates on a structurally distinct group** under an independent architecture and evaluation protocol.

We report all statistics with explicit caveats, including small sample sizes at the seed level ($n=5$) and candidate-pair level (16–26 pairs per split for $S_4$), an explicit statement of our SD convention, a formal power argument for the exact-zero result, a pre-registration statement for seed selection, and a rare (but non-zero) exception to metric-equality that we flag rather than over-explain.

**Keywords:** grokking, mechanistic interpretability, group theory, weight decay, generalization, functional probes, numerical precision, closed-domain verification, multi-seed statistics, pre-registration