GDN Layers are Freeze-Tolerant: Efficient SFT Strategies for Hybrid Gated Delta Network Architectures

We present a freeze-sensitivity study of supervised fine-tuning in a hybrid GDN-Attention language model, using Qwen3.5-0.8B as the testbed. We compare 7 freeze patterns across 2 downstream tasks with 6 random seeds each (84 experiments total).

Our main finding is that GDN layers are highly freeze-tolerant. Freezing all 18 GDN layers (51.6% of parameters) yields performance statistically indistinguishable from freezing the 6 Gated Attention layers (14.6%), while significantly outperforming position-based freezing of comparable depth. We also find that selective freezing can outperform full SFT on knowledge QA, suggesting a possible regularization effect, and that attention projection parameters are disproportionately important for knowledge adaptation.

Overall, the results suggest a practical guideline for fine-tuning hybrid GDN-Attention models: freezing GDN layers preserves performance surprisingly well while reducing training cost. This record contains the preprint PDF.