Does the reward matter? A placebo trial for zero-data self-play

Abstract

Recent work on zero-data self-play suggests that a language model may improve at competition math by inventing and solving its own small coding puzzles without human-written data. Separate studies show that reinforcement learning can improve math scores even when rewards are superficial and evaluate only output formatting. Is a self-trained coding model learning from verifying its code, or merely producing answers inside the expected tags? We test this with a placebo-controlled solver trial using fixed seed tasks from Absolute Zero (Qwen2.5-Coder-3B). We hold the base model, the 100-step coding task stream, and the optimizer fixed while varying the reward function (plus the rollout seed on a second treatment run). A placebo reward gives full credit whenever a response includes the required <think> and <answer> tags (plus a code block on function tasks), without running the code or checking the answer. The real reward executes the Python code and pays only when the output is right. Across six math benchmarks, the placebo model learns the tag format within 15 steps and shows no detectable gain, finishing 1.9 percentage points below the base model (95% CI [−4.9, 0.6]). The execution-checked reward gains 5.7 and 5.1 points across two seeds (with seed 2 scored on Absolute Zero's prompt only). Their mean gain recovers about 72% of the improvement seen in the released model and beats the placebo by about 7 points. In this 100-step cross-domain setting from code to math, checking answer correctness drives the S6 improvement where a tag-format reward does not.

Main Results (S6, greedy decoding, judge J2, percentage points)

Files

• zero-placebo.pdf: the paper.
• fig0_schematic.png, fig1_teaser.png, fig2_templates.png, fig3_dynamics.png: Figures 1–4.
• zero-placebo-v1.0.zip: the paper (PDF and LaTeX source), figures and figure script, preregistrations, evaluation and training scripts, committed result JSONs, README, licences, and SHA256SUMS.

Licence: CC BY 4.0; code scripts are additionally offered under the MIT License. The paper is AI-assisted; its section "Use of AI tools" states how. Not yet peer reviewed.