There is a newer version of the record available.

Published March 5, 2026 | Version v4

Small Models Can Learn Complex Behaviors — They Just Need the Right Examples

Authors/Creators

Description

Contrastive behavioral pairs injected into 5% of pretraining blocks break the small-model behavioral emergence wall. A 7M-parameter model achieves bias and sycophancy scores exceeding vanilla 34M, at zero additional inference cost.

Files

contrastive_pretraining.pdf

Files (158.6 kB)

Name Size
md5:3e00b1833a69b23912e414019ad7a5df
158.6 kB Preview Download