Weight Decay as a Necessary Condition for Grokking: Hierarchical Structure Learning in Group Multiplication Tasks

please refer to new version v1.8 (10.5281/zenodo.21204367) 

# Abstract

We investigate the role of weight decay in the "grokking" phenomenon, where neural networks achieve perfect training accuracy long before generalizing. Through systematic experiments on the dihedral group $D_{12}$ multiplication task, we report three findings:

1. **Weight decay is necessary, not optional**: With weight decay disabled (WD=0), networks achieve perfect training accuracy but fail to generalize (Val Acc < 15%) even after 25,000 training steps. With WD=0.2, the same networks reach 100% validation accuracy by step 16,000.

2. **Hierarchical structure emerges under compression**: When weight decay is active, networks learn the coarse-grained quotient structure ($\mathbb{Z}_2$ parity, 90% accuracy at step 500) hundreds of steps before element-level prediction (near 0%). This hierarchical progression is absent without weight decay.

3. **Synchronized phase transition at step ~22,000**: All seeds exhibit a brief generalization collapse around step 22,000, driven by weight decay's cumulative compression reaching a critical threshold. This collapse disappears entirely when WD=0, confirming its regularization-induced origin.

We also demonstrate that **functional metrics (Cycle Closure Count) reliably track generalization, while topological metrics (persistent homology $H_1$) do not**. These results frame grokking as a compression-driven phenomenon where regularization forces hierarchical structure extraction.