SpectralBERT: Accelerating Context Windows via Spectral Attention Dynamics

The quadratic time and memory complexity of standard self-attention mechanisms (O(N2)) imposes fundamental computational
limits on scaling Transformer context windows. In this work, we introduce SpectralBERT, an efficient bidirectional
architecture that replaces standard pairwise token attention with a fixed-size spectral memory mechanism. By projecting
sequence embeddings into the frequency domain via Real Fast Fourier Transform (RFFT), adaptively pooling frequency
coefficients into compact slot representations, and retrieving context via memory-guided attention, SpectralBERT achieves O(N
log N) time complexity with constant sequence memory overhead (O(1)) during feature mixing. Empirical evaluations on
WikiText-2 demonstrate a final Masked Language Modeling (MLM) loss of 5.6420, outperforming a comparably sized BERT-
Mini baseline (loss 6.5773) by 14.3% while training 1.13x faster. Pre-training on WikiText-103 at an 8,192-token context
length yields a loss of 5.2525 (Perplexity = 191.05). Forward execution scaling benchmarks confirm a 14.51x throughput
speedup at a 65,536-token sequence length on a single NVIDIA Tesla T4 GPU. These findings demonstrate that frequency-
domain memory structures offer a scalable and mathematically sound alternative to quadratic self-attention for long-context
language modeling.