FELA: Fourier Encoder with Linear Attention for Efficient Long Context Language Modeling

Transformers suffer from inherent compute/memory bottlenecks such as quadratic prefill and linear cache size at long sequence lengths. In this work we introduce FELA, an architecture with constant decode state size regardless of context length. This model blends three sequence mixers together: causal spectral convolution, gated delta-recurrence, and local sliding window attention. Under a rigorous matched token and parameter budget experiment, FELA matches or leads FlashAttention-2 transformers on validation loss, achieving a 0.011 bpb improvement at 700M parameters while reducing decode memory usage by up to 2 orders of magnitude. We ablate each of FELA’s mixer components and show they are each required for FELA’s cross entropy performance. FELA can run directly using standard PyTorch calls on a CPU, without specialized custom kernels. We benchmark these models and qualitatively show the well known trade-offs of bounded state models, such as decreased recall when compared to a transformer with standard attention. The architecture naturally extends to untokenized byte inputs as well since there is more memory budget due to our efficiency.