HON: A Harmonic Oscillator Network for Temporal Memory in Sequence Modeling

We present HON (Harmonic Oscillator Network), a linear hybrid sequence modeling architecture
inspired by the physics of damped harmonic oscillators. The architecture integrates three parallel functional
branches within each block:
1. LACT (Local Adaptive Causal Transform)  A content-dependent multi-kernel causal convolution
for capturing local n-gram patterns.
2. HON (Harmonic Oscillator State Space) A continuous-time temporal memory layer parameteriz‐
ing damped harmonic oscillations evaluated via FFT-based causal convolutions.
3. ASSOC (Linear Associative Memory)  A normalized linear attention mechanism operating via fast
cumulative prefix sums.
These representations are combined using a dynamic learnable gating mechanism. Initial evaluations on
WikiText-2 demonstrate an 18.5% improvement in validation perplexity compared to an equivalent
Transformer baseline under identical parameter budgets. Additional experiments on Tiny ImageNet show
that the architecture successfully learns 2D spatial representations formatted as 1D sequences.
We do not claim that HON replaces Self-Attention or outperforms state-of-the-art models on large-scale
benchmarks. The primary goal of this technical report is to document an initial experiment and share the
findings with the open-source community. We also acknowledge significant limitations in our current im‐
plementation, including generation quality, computational overhead, and the need for more careful fusion
mechanisms.