HSNNM: A Sparse, Dynamically Routed Feed-Forward Alternative for Transformer Layers

We describe the Hebbian Sparse Neural Network Model (HSNNM), a transformer feed-

forward layer that replaces the standard dense sublayer with a set of sparse, competitively routed

sub-networks (”lobes”). The design combines top-1 lobe routing, k-Winners-Take-All (k-WTA)

intra-lobe sparsity, a local Hebbian strength-modulation rule, and a net2net-style structural

growth mechanism. On a 7.33M-parameter model trained on a 10% slice of TinyStories, HSNNM

reduces FLOPs per token by 49.6% relative to a matched dense baseline (7.18M vs. 14.25M

FLOPs/token) at a cost of 3.4 percentage points of validation accuracy (76.06% vs. 79.45%).

Ablations over a 5,000-step stress test indicate that both the Hebbian mechanism and k-WTA

sparsity are associated with somewhat lower catastrophic forgetting than variants without them,

though part of this comparison uses a different evaluation protocol from the main results and

should be treated as suggestive rather than conclusive. We report these results, along with

their limitations, as a data point on the efficiency-accuracy trade-off available from a compute-

skipping sparse FFN design at small scale.