There is a newer version of the record available.

Published March 5, 2026 | Version v5

Small Models Can Learn Complex Behaviors — They Just Need the Right Examples

Authors/Creators

Description

Contrastive behavioral pairs injected into 5% of pretraining blocks break the small-model behavioral emergence wall. A 7M-parameter model achieves bias and sycophancy scores exceeding vanilla 34M, at zero additional inference cost.

Files

contrastive_pretraining.pdf

Files (158.6 kB)

Name Size
md5:53370ec39426010dd762fe2d0e909f86
158.6 kB Preview Download