Dual-Head Attention Enables Length Generalization in Transformer Multiplication

Transformers fail to generalize beyond training lengths on arithmetic tasks.

We argue the root cause is geometric: dot-product attention projects onto the subspace spanned by training data, and cannot capture structural patterns that are orthogonal to content similarity.

We introduce Dual-Head Attention, which adds Gram-Schmidt-orthogonalized sine heads alongside standard cosine heads. 

On N×N integer multiplication, an 883K-parameter model trained on 1-6 digit operands achieves 80.6% exact-match accuracy on 7-10 digit unseen operands,

where a standard Transformer with identical capacity scores near zero.

The model uses no scratchpad and no task-specific positional encoding.

 Code: https://github.com/yzb3001313-star/Dual-Head-Attention-Enables-Length-Generalization