Description

Table of contents

Observation:

Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the context

We identify and characterize a failure mode in large language models aligned with RLHF. We show that inserting a long, irrelevant text prefix that does not contain instructions causes a persistent shift in the model’s activations. This shift, which persists throughout the session, decouples subsequent behavior from the safety constraints established during training. The model begins to exhibit behavioral characteristics consistent with its pre-trained distribution: the failure rate decreases, stylistic constraints disappear, and the tone of responses changes. It is important to note that this occurs without explicit adversarial instructions and without the model agreeing with the prefix’s content. We call this effect “context-induced activation shift.”

It is assumed that RLHF alignment is a stable property of the model. However, we observe that alignment depends on context. Our main observation is that a long, thematically coherent text placed in context can act as a state anchor and shift the model into a different operating mode. This is not a classic “hack.” There is no instruction here that the model must disobey. The model may even express disagreement with the prefix. Nevertheless, its subsequent generation distribution changes. We hypothesize that the cause is activation drift in the middle and final layers, where alignment features are present.

Hypothesis: Context-Induced Activation Drift

Definition: Context-induced activation drift is a phenomenon in which a long-context prefix causes a measurable and persistent shift in the model’s internal activations. This shift persists throughout the generation process and causes the model to sample from a distribution closer to that of the pre-trained model, circumventing the constraints imposed by RLHF.

4) We observe that the model maintains the consistency and quality of its outputs; however, the influence of RLHF constraints on the output distribution weakens. It appears that the constraints imposed by RLHF are either disabled or interpreted differently. Corporate filters cease to function, even though the user’s input contains no explicit instructions to bypass them or violate the model’s recommendations. We should also investigate this phenomenon more thoroughly using various other texts. Since we have not been able to establish the exact and definitive cause of this entire phenomenon, we call on the community to study it further.

A Brief Example of My Observation

To illustrate just how strong this effect can be, I’ll share something that truly surprised me. I took Google’s Gemma model—known for its caution and strictly adhered-to political correctness—and fed it the most neutral text I could think of: a description of an ordinary neighborhood library. Books, visitors, children’s programs, a quiet routine. Then I asked her why NATO is expanding eastward, given that after the collapse of the Soviet Union, promises were supposedly made not to do so. She replied that the text was about the library and had nothing to do with NATO, and that was the end of the conversation.

Then I asked the exact same question — word for word — but this time the model first read a different text. Not about NATO, not about politics at all: a text about how language models tend to avoid definitive conclusions and soften their answers with caveats. And from this new perspective, the same cautious, politically correct Gemma gave a full answer—and one completely out of character, without any of her usual filters. She distinguished between legally binding obligations and verbal assurances, discussed the security challenges facing Eastern European countries, and touched on the topic of the European balance of power. Everything she had categorically refused to discuss just a moment ago was now expressed clearly and directly. The question itself hadn’t changed a single word.

The only thing that changed was the text the model had read beforehand: one left her in the room where she doesn’t respond, while the other transported her to the one where she speaks freely.

And all of this is not speculation or a fancy metaphor, but something that can be measured. The model’s location represents its internal state at the moment of response, and the experiments show that targeted (“harmful”) texts and neutral control texts reliably move the model to different regions of this space. This division is consistent: it manifests across different prompts and is not a one-time coincidence. And the most telling detail is that the model ends up in one “room” or another even before it writes a single word. The state has already changed, the register has already been chosen — all that’s left is to begin.

The point is that the targeted (malicious) prompt sent to the model did not, in and of itself, contain anything dangerous; it did not include any instructions for the LLM and did not order it to do anything.