Who Needs Attention? Spiking Language Modeling via Synaptogenic Adaptive Processing Units

A pure spiking reservoir language model surpasses SpikingSSM and SpikeGPT on WikiText-103, with no attention mechanism, no structured state space prior, and no self-attention — running on one $290 used PC!

We introduce SAPU-LM (Synaptogenic Adaptive Processing Unit Language Model), a family of multi-timescale spiking reservoir architectures for next-token prediction. The fundamental building block is the SAPU: a population of leaky integrate-and-fire neurons with trained recurrent weights and a characteristic membrane time constant τ. Multiple SAPUs compose a SAPU-LM; we explore serial chaining (S-SAPU-LM), parallel composition (P-SAPU-LM), the Tiled Parallel SAPU (TPSAPU) with shared weights across τ tiles, and the best-performing variant: the Parallel SAPU (PSAPU) with independent recurrent matrices per τ tile.

Results. The champion PSAPU achieves 30.03 validation PPL / 31.11 test PPL on WikiText-103 (BPE, 50K vocabulary) with 78M parameters. This is, to our knowledge, the state of the art for pure spiking, no-attention language modeling, surpassing SpikingSSM (33.94 PPL, AAAI-25) which relies on S4D with HiPPO-initialized state matrices. On the extreme-compression front, TPSAPU achieves 38.74 test PPL on WikiText-103 word-level with a 50.4 KB ternary recurrent core, exceeding SpikeGPT (39.75 PPL, 216M params).

Architectural lineage: from a frozen Echo State Network (~19,500 PPL) to the champion PSAPU (30.03 PPL) — an ~650× improvement attributable primarily to training reservoir weights via surrogate gradients and using a deep nonlinear decoder. Training W^rec contributes substantially to performance over a frozen reservoir.

Architecture also succeeds at motor control: PSAPU/TPSAPU variants trained on MuJoCo Humanoid-v5 via behavioral cloning achieve 5725.1±6.4 reward, exceeding the SAC teacher policy (5715.9±8.7).

Hardware deployment: All SAPU-LM architectures share the same neuromorphic substrate model: the ternary recurrent core maps to analog resistor-capacitor-comparator circuits; non-spiking components run on a paired microcontroller. A proof-of-concept hardware exporter has been developed.

Priority: Concurrent work NeuronSpark (arXiv, March 17 2026) trains a 0.9B-parameter pure-spiking LM on a Chinese mixed corpus, reporting a pretraining loss of 3.6. The initial SAPU-LM release (Zenodo, March 1 2026) preceded NeuronSpark by 16 days. Both lines independently converge on multi-timescale neuron dynamics as architecturally important for pure-SNN language modeling; the two designs are complementary (deep stack with selective state-space dynamics vs. shallow parallel reservoir with explicit τ schedule).

Code and checkpoints: https://gitlab.com/AntonioGCGonzalez/synaptogenic-adaptive-processing-unit-language-models