Kernels Might Be What You Need: Efficient Sequence Modeling with K-Operators

We introduce K-Operators, a kernel-decomposed sequence modeling architecture that

replaces attention entirely with structured causal kernel operators. On Tiny Shakespeare

character-level modeling, a 1.14M-parameter K-Operators model achieves 4.43 ±0.05 validation

perplexity across 7 seeds—approaching the 4.35 PPL of a 10.65M-parameter Transformer baseline

(nanoGPT) while using 9.3×fewer parameters and requiring no positional encodings.

The architecture decomposes sequence mixing into a hierarchy of operators: K1 layers for

position-wise feature mixing, K2 layers for causal sequence interaction via a learned base kernel

combined with low-rank gamma-decayed recurrence, and a K0 layer for final rescaling. These are

composed into a K-Stack backbone (K1 →K(×N )

2 →K1 →K0) and refined through a learned

iterative equilibrium loop governed by a scalar step-size η. Two interchangeable gamma-decay

backends (mask and block) offer different memory/speed trade-offs. Diagnostic analysis reveals

interpretable learned dynamics: the model progressively transfers sequence mixing from the

initialized base kernel to the adaptive recurrent path, develops per-layer functional specialization,

and learns to self-regulate the refinement loop—including robustness to 10×learning rate

misspecification via automatic η suppression.