STRIKETHROUGH: In-Context Retraction Tags Mitigate Self-Conditioning in Language Models

Recent work shows that autoregressive language models "self-condition": when
their context contains their own prior mistakes, they become more likely to
produce new ones, and this does not disappear with model scale for
non-thinking models. The standard mitigation is to delete erroneous steps from
the context. We test the opposite operator: keep the erroneous step but wrap
it in an explicit <retracted> tag, so the model conditions on it as a labeled
negative example rather than a fluent pattern to continue.

On a controlled running-sum task with a fixed history and an injected error
rate, we find that (i) self-conditioning reproduces on small open models
(Gemma-3 4B, Qwen-2.5 7B); (ii) a 2x2 ablation isolates the retraction tag --
not an accompanying answer hint -- as the active ingredient; (iii) a recall
probe shows that tagging preserves information deletion destroys, improving
recall of a discarded value by roughly 16x; and (iv) the effect strengthens
with model capability: on the 7B model the tag alone nearly eliminates
self-conditioning, whereas the 4B model requires the tag together with a
verified anchor.

All experiments are seed-locked and reproducible on local Ollama models. Code
included. We discuss limitations and the natural next test on thinking-capable
models.