Adaptive Repetition Suppression in Language Models via Learned Risk Prediction- Field-Separated Cognitive Architectures (FSCA)

 We present an empirical study of repetition degeneration in large language models and introduce a decode-time intervention method that reduces repetitive text generation while preserving output coherence. Our approach trains a lightweight risk predictor on model hidden states to estimate the probability that the next generated token will repeat a recently produced token. The predictor achieves F1 > 0.96 with up to 80× separation between high-risk and low-risk positions.

Unlike attention-modification approaches, which we systematically evaluate and show to fail on pretrained models due to training–inference mismatch and compensation effects, our method intervenes only at the sampling stage. The base model’s architecture and attention patterns remain unchanged. When the predicted risk exceeds a threshold, recently generated tokens are adaptively penalized during sampling.

Across multiple open-ended generation prompts, the proposed method reduces repetition rates by approximately 48% and increases lexical diversity (Distinct-2) by approximately 17% relative to a no-penalty baseline, without introducing incoherence or degraded fluency.

In addition to the working system, this release documents five failed attention-gating approaches and analyzes the mechanisms underlying their failure. These negative results suggest that modifying attention dynamics in pretrained language models via adapters is structurally unstable, whereas decode-time intervention based on anticipatory signals in hidden states is robust.

This work provides empirical evidence that pretrained language models encode predictive signals of their own generation failure modes in hidden representations and that these signals can be exploited for adaptive, non-invasive control at inference time.