Published March 5, 2026
| Version v3
Journal article
Open
Small Models Can Learn Complex Behaviors — They Just Need the Right Examples
Authors/Creators
Description
A 7M-parameter language model scores rho=0 on bias and sycophancy — behaviors that emerge at 18M-34M in vanilla training. We show this is a data problem, not a capacity problem. Injecting contrastive behavioral pairs into 5% of training blocks (0.11% of total tokens) breaks the wall: bias reaches rho=0.433 and sycophancy rho=0.513, exceeding vanilla 34M sycophancy at 5x fewer parameters and zero additional inference cost.
Files
contrastive_pretraining.pdf
Additional details
Related works
- Is supplemented by
- 10.5281/zenodo.18854944 (DOI)
- 10.5281/zenodo.18865862 (DOI)
- 10.5281/zenodo.18865199 (DOI)