SuperBake: Installing Verified Facts into Transformer Weights by Direct Construction

We present SuperBake, a system that installs new factual knowledge into the weights of

large language models without any gradient steps. Instead of fine-tuning, low-rank adaptation,

or retrieval, SuperBake measures how the stock model represents each question and then hand-

constructs a small circuit of neurons — exact weights, written directly — in a region appended

to the network’s MLPs. Each installed fact receives a physical address (layer and slot coordi-

nates), is behaviorally verified across phrasing variants, and is neutralized (zeroed) if verification

fails, so delivered weights never carry silently broken knowledge. The output is a standard check-

point that stock inference code loads unchanged, together with a receipt listing every fact, its

verification status, and its coordinates.

On a 7,450-row battery over 1,000 facts (Llama-3.1-8B-Instruct), the constructed engine veri-

fies 91.3% of rows overall against 76.9% for a strong masked-SGD baseline — while leaving prose

perplexity at the stock baseline, where the SGD baseline more than doubles it. Constructed

facts answer reverse questions at∼94% where the trained baseline scores 0.7%: the reversal

curse does not apply when the reverse mapping is simply written. Delivered, stock-loadable

checkpoints reach 93–97% primary-question recall on Llama-3.1-8B and Qwen2.5-7B. Construc-

tion is fast: measured end-to-end bakes run in minutes on one 80 GB GPU, and a reduced

pipeline wrote 192 facts into a raw Pythia-6.9B at 14.8 facts per second.

The method was derived from a mechanistic post-mortem of what stochastic gradient descent

actually builds when it crams facts into appended neurons. We report those findings: dense fact

storage lands in a coherent sub-threshold population code living in activation-function leakage

— the same signal that damages prose — and several transport and capacity laws that any

weight-space editor must respect. SuperBake is the constructive rebuild of that mechanism: the

same associative memory, but gated, addressable, verifiable, and harmless to the host model.