DAWN: Dynamic Subspace Composition for Structural Interpretability

DAWN replaces static weight matrices with dynamic subspace composition over shared neuron pools. Each token dynamically constructs its own projection matrix by selecting sparse neuron subsets, enabling structural interpretability through traceable routing paths.

With 39M parameters, DAWN achieves comparable performance to a 40M vanilla Transformer (perplexity 42.2 vs 41.2) on C4 (5B tokens) while providing interpretable internal structure. Analysis reveals emergent neuron specialization: Query/Key neurons develop distinct roles (correlation r=-0.75), and neurons align with part-of-speech categories without explicit supervision.

This upload contains the paper (PDF) and pretrained model weights (PyTorch).

Code: https://github.com/madst0614/DAWN