There is a newer version of the record available.

Published March 5, 2026 | Version v3

Small Models Can Learn Complex Behaviors — They Just Need the Right Examples

Authors/Creators

Description

A 7M-parameter language model scores rho=0 on bias and sycophancy — behaviors that emerge at 18M-34M in vanilla training. We show this is a data problem, not a capacity problem. Injecting contrastive behavioral pairs into 5% of training blocks (0.11% of total tokens) breaks the wall: bias reaches rho=0.433 and sycophancy rho=0.513, exceeding vanilla 34M sycophancy at 5x fewer parameters and zero additional inference cost.

Files

contrastive_pretraining.pdf

Files (232.4 kB)

Name Size
md5:0bb5da1bc49db8119f08391545d9e665
232.4 kB Preview Download

Additional details