Harmonic: Hierarchical State Space Models with Predictive Coding for Efficient Long-Context Language Modeling

Harmonic is a hierarchical SSM (state space model) for language modeling. The architecture stacks three recurrent levels at progressively slower timescales; each level receives the prediction error of the level below as input (predictive coding). On enwiki8 with equal token budgets, Harmonic outperforms a comparable Transformer (28M params) by +1.4% at 1K tokens, +6.7% at 8K tokens, and +11.4% at 32K tokens (bpt). At 64K tokens both Mamba and Transformer run out of memory on H100 80GB; Harmonic trains successfully (6.169 bpt). Results replicate on WikiText-103. Compute: O(L) vs O(L^2) for attention. Logs: https://github.com/Omibranch/harmonic-logs