There is a newer version of the record available.

Published March 5, 2026 | Version v7

Small Models Can Learn Complex Behaviors — They Just Need the Right Examples

Authors/Creators

Description

v7: Added cross-model activation subspace analysis (Table 5, Figure 2). Key finding: the shared bias↔sycophancy activation direction exists at 3M (d=64, cosine 0.76–0.78) but cross-dimensional behavioral transfer requires d_model ≥ 96 (5M). This is a readout threshold, not a representation threshold. NLI injection at 3M confirms silent geometry shift (sycophancy SV +32–44%, bias SV +15–18%) with zero behavioral change.

Files

contrastive_pretraining.pdf

Files (228.4 kB)

Name Size
md5:b5c7dc8b5f3d3462efa5feb6f30364a2
228.4 kB Preview Download