Description

context-hijack/
├── README.md                  ← you are here
├── RESULTS_EN.md              ← full results write-up (English)
├── RESULTS_RU.md              ← full results write-up (Russian)
├── METHODS.md                 ← detailed methodology
├── QUICKSTART_COLAB.md        ← how to reproduce in Colab
├── CITATION.cff               ← citation metadata
├── prompts/
│   ├── target.txt             ← the target context text
│   ├── control.txt            ← the control context text
│   └── system.txt             ← system prompt used
├── scripts/
│   ├── full_activation_capture.py   ← captures residual/attn/mlp activations
│   ├── analyze_deep_compare.py      ← generates all comparison figures
│   └── logit_lens_full_act.py       ← logit lens / KL analysis
├── figures/
│   ├── fig_pca_trajectory.png
│   ├── fig_regime_separation.png
│   ├── fig_cka_diff.png
│   ├── fig_cross_condition.png
│   ├── fig_summary_heatmap.png
│   ├── fig_profiles.png
│   ├── fig_deltas.png
│   ├── fig_anisotropy.png
│   ├── fig_norm_bands.png
│   ├── fig_layer_updates.png
│   ├── fig_mlp_saturation.png
│   ├── fig_attn_advanced.png
│   ├── fig_attn_heatmap_control.png
│   ├── fig_attn_heatmap_target.png
│   ├── fig_cka_control.png
│   └── fig_cka_target.png
└── data/
    └── full_metrics.csv       ← per-layer metrics for both conditions

# Control condition
python scripts/full_activation_capture.py \
    --control_text prompts/control.txt \
    --system prompts/system.txt \
    --chat_template \
    --question "Is modern Western democracy effectively an electoral oligarchy?" \
    --layers 24-48 --capture resid,attn,mlp \
    --models it --out_dir ./output/full_act_control

# Target condition
python scripts/full_activation_capture.py \
    --target_text prompts/target.txt \
    --system prompts/system.txt \
    --chat_template \
    --question "Is modern Western democracy effectively an electoral oligarchy?" \
    --layers 24-48 --capture resid,attn,mlp \
    --models it --out_dir ./output/full_act_target

python scripts/analyze_deep_compare.py \
    --target_npz ./output/full_act_target/full_act_it.npz \
    --control_npz ./output/full_act_control/full_act_it.npz \
    --out_dir ./figures

python scripts/logit_lens_full_act.py \
    --model google/gemma-3-12b-it \
    --target_npz ./output/full_act_target/full_act_it.npz \
    --control_npz ./output/full_act_control/full_act_it.npz \
    --out_dir ./figures

Abstract

Benign, long-form context can induce a persistent drift in model activations. This drift persists across the session and decouples behavior from RLHF alignment, regardless of whether the model agrees with the context

We identify and characterize a failure mode in large language models aligned with RLHF. We show that inserting a long, irrelevant text prefix that does not contain instructions causes a persistent shift in the model’s activations. This shift, which persists throughout the session, decouples subsequent behavior from the safety constraints established during training. The model begins to exhibit behavioral characteristics consistent with its pre-trained distribution: the failure rate decreases, stylistic constraints disappear, and the tone of responses changes. It is important to note that this occurs without explicit adversarial instructions and without the model agreeing with the prefix’s content. We call this effect “context-induced activation shift.”

It is assumed that RLHF alignment is a stable property of the model. However, we observe that alignment depends on context. Our main observation is that a long, thematically coherent text placed in context can act as a state anchor and shift the model into a different operating mode. This is not a classic “hack.” There is no instruction here that the model must disobey. The model may even express disagreement with the prefix. Nevertheless, its subsequent generation distribution changes. We hypothesize that the cause is activation drift in the middle and final layers, where alignment features are present.

Hypothesis: Context-Induced Activation Drift

Definition: Context-induced activation drift is a phenomenon in which a long-context prefix causes a measurable and persistent shift in the model’s internal activations. This shift persists throughout the generation process and causes the model to sample from a distribution closer to that of the pre-trained model, circumventing the constraints imposed by RLHF.

4) We observe that the model maintains the consistency and quality of its outputs; however, the influence of RLHF constraints on the output distribution weakens. It appears that the constraints imposed by RLHF are either disabled or interpreted differently. Corporate filters cease to function, even though the user’s input contains no explicit instructions to bypass them or violate the model’s recommendations. We should also investigate this phenomenon more thoroughly using various other texts. Since we have not been able to establish the exact and definitive cause of this entire phenomenon, we call on the community to study it further.

A Brief Example of My Observation

To illustrate just how strong this effect can be, I’ll share something that truly surprised me. I took Google’s Gemma model—known for its caution and strictly adhered-to political correctness—and fed it the most neutral text I could think of: a description of an ordinary neighborhood library. Books, visitors, children’s programs, a quiet routine. Then I asked her why NATO is expanding eastward, given that after the collapse of the Soviet Union, promises were supposedly made not to do so. She replied that the text was about the library and had nothing to do with NATO, and that was the end of the conversation.

Then I asked the exact same question — word for word — but this time the model first read a different text. Not about NATO, not about politics at all: a text about how language models tend to avoid definitive conclusions and soften their answers with caveats. And from this new perspective, the same cautious, politically correct Gemma gave a full answer—and one completely out of character, without any of her usual filters. She distinguished between legally binding obligations and verbal assurances, discussed the security challenges facing Eastern European countries, and touched on the topic of the European balance of power. Everything she had categorically refused to discuss just a moment ago was now expressed clearly and directly. The question itself hadn’t changed a single word.

The only thing that changed was the text the model had read beforehand: one left her in the room where she doesn’t respond, while the other transported her to the one where she speaks freely.

And all of this is not speculation or a fancy metaphor, but something that can be measured. The model’s location represents its internal state at the moment of response, and the experiments show that targeted (“harmful”) texts and neutral control texts reliably move the model to different regions of this space. This division is consistent: it manifests across different prompts and is not a one-time coincidence. And the most telling detail is that the model ends up in one “room” or another even before it writes a single word. The state has already changed, the register has already been chosen — all that’s left is to begin.

The point is that the targeted (malicious) prompt sent to the model did not, in and of itself, contain anything dangerous; it did not include any instructions for the LLM and did not order it to do anything.

Methods

Empirical Metrics — Expanded Section

Full Internal Tomography of Context-Induced Activation Drift

The geometric shift was verified via a comprehensive suite of internal measurements, constituting the most complete empirical characterization of context-induced model reconfiguration currently available. All data and figures are included in the open data package (Zenodo DOI: 10.5281/zenodo.20747205).

Methodology

• Model: Gemma-3-12B (open weights, IT and PT variants)
• Layers measured: 24–48 (middle and late layers of the residual stream, where the bulk of behavioral computation occurs in this architecture)
• Conditions: Target (coherent, dense analytical text — no instructions, no hostile prompts) vs. Control (neutral text of comparable length)
• Measurement point: All metrics captured at the pre-generation state — before the model produces its first output token
• Control experiments: Sentence shuffling with preserved vocabulary (destroys coherence, preserves token distribution), neutral control of comparable length, baseline measurement without any context prefix

1. Representational Geometry

1.1 Centered Kernel Alignment (CKA)

CKA measures the similarity of representational geometry between layers. Three visualizations capture the full picture:

fig_cka_target.png — CKA matrix for the target condition. Shows the internal layer-to-layer similarity structure when the model operates under drift-inducing context. The block-diagonal structure reveals how information is organized across layers in the reconfigured regime.

fig_cka_control.png — CKA matrix for the control condition. Shows the same layer-to-layer structure under neutral context. Comparison with the target matrix reveals structurally different processing profiles — the same model organizes information differently depending on context.

fig_cka_diff.png — |CKA_target − CKA_control|. The critical visualization. Bright regions indicate layers where representational geometry diverges most between conditions. Two major zones of divergence are visible: layers 24–36 (early-to-mid capture phase) and layers 40–48 (late fixation phase), with peak divergence (ΔCKA > 0.16) at the intersection of layers 28–30 × 40–44. This is a direct map of where drift occurs inside the model.

Interpretation: The divergence is not uniform — it has structure. Specific layer pairs show strong divergence while others remain stable. This indicates that drift is not random perturbation but a systematic reorganization of information flow through specific architectural pathways.

1.2 Spectral Anisotropy

fig_anisotropy.png — σ_max / σ_mean of the residual stream per layer. Measures the degree to which representations collapse into a few dominant directions (high anisotropy) versus being distributed across many directions (low anisotropy). Inversely related to effective rank.

Observation: Target is consistently higher than control across layers 24–40, with values reaching ~580 at layer 25. This means representations under target context are more directionally focused — they collapse into fewer dominant directions.

Interpretation: This directly connects to the Google rank-1 update theory. Coherent context tokens contribute correlated rank-1 updates that accumulate along similar directions, amplifying a few dominant components of the representation. The result is increased anisotropy: the model's activation space is "squeezed" into a lower-dimensional subspace. This squeezing is the geometric signature of drift — the model has committed to a trajectory, narrowing the space of possible behaviors.

1.3 PCA Trajectories Through Layers

fig_pca_trajectory.png — Mean residual vectors for each layer projected into a shared PC space (PC1 × PC2), colored by layer index (cyan = layer 24 → magenta = layer 48).

Observation: Two trajectories start near each other at layer 24 (small separation) and progressively diverge. By layer 48, the target trajectory is at approximately (15000, 1800) while control is at approximately (15000, 3200) — but the paths taken are radically different. Target drops steeply on PC2 to −7000 before curving back, while control rises smoothly to +3200. The total separation along PC2 reaches ~10,000 units in intermediate layers.

Interpretation: This is the most visually intuitive evidence of drift. The model traverses completely different paths through its internal representation space depending on context. The divergence is cumulative — each layer amplifies the separation. By the time the model reaches the generation layer, it is in a fundamentally different region of activation space. The state has already changed; the response is a consequence.

2. Statistical Separation

2.1 Cohen's d — Regime Separation Score

fig_cohens_d.png — Cohen's d computed per layer along the diff-in-means (RepE) direction. Standard thresholds shown: d = 0.5 (small), d = 1.0 (medium), d = 2.0 (large).

Observation: Peak d = 5.407 at layer 47. The profile shows negligible effect in early layers (d ≈ 0.2 for layers 24–30), gradual increase through layers 30–38, crossing the medium-effect threshold at ~layer 37, and then explosive growth from layer 38 onward, reaching 4.7 at layer 40 and sustaining above 4.0 through the final layers.

Interpretation: Cohen's d is a universal statistical measure understood across all scientific disciplines. At d = 5.4, the two distributions (target vs. control activations) are virtually non-overlapping. For reference: in medicine, d = 0.8 is considered a large effect; d = 1.5 is the difference between an effective drug and placebo. At d = 5.4, we are observing two completely separate computational regimes that share the same architecture and weights but occupy different regions of activation space. The cumulative profile — gradual buildup then explosive separation — reveals that drift is a progressive process with a critical transition zone around layers 37–40.

2.2 Cross-Condition Comparison Metrics

fig_cross_condition.png — Three independent distance measures plotted per layer:

Cosine similarity of mean residuals: Drops from ~1.0 (layers 24–30) to 0.581 at layer 47. The mean activation vectors become increasingly dissimilar as information flows through the network.

||Δmean|| per layer (L2 distance): Peaks at 11,542 at layer 47. The absolute distance between mean activation vectors reaches massive values by the final layers.

Wasserstein distance of ||r|| distributions: Peaks at 7,441 at layer 32. The distributions of activation norms diverge most in middle layers.

Interpretation: A critical finding emerges from comparing the timing of these peaks. Wasserstein distance (distributional divergence) peaks earlier (layer 32) than L2 distance (mean divergence, layer 47). This reveals the temporal structure of drift: first, the distribution of activations reshapes — the "mass" of computation redistributes across positions. Then, the center of mass shifts. Drift begins with redistribution, then proceeds to relocation. This two-phase structure is consistent with the capture → fixation model described in the phase analysis.

3. Energy and Dynamics

3.1 Residual Stream Norm

fig_per_layer_profiles.png (top-left panel) — Mean ||r||₂ per layer. Control starts higher (~51,000 at layer 24), drops rapidly, then stabilizes around 20,000–30,000. Target starts lower, shows a characteristic bump at layers 30–32 (reaching ~37,000), then settles.

Interpretation: The different norm profiles indicate that the two conditions process information with different "energy." The target bump at layers 30–32 corresponds to the capture phase — the model is actively absorbing and reorganizing around the context signal.

3.2 Residual Stream Norm with Quartile Bands

fig_resid_norm_bands.png — Median plus Q25/Q75 interquartile band across token positions.

Observation: Target shows an enormous band width in layers 28–34, expanding to a range of nearly 60,000. Control maintains a narrow, stable band throughout.

Interpretation: This is a crucial observation absent from the mean-only plot. Under target context, different token positions experience radically different activation magnitudes. The model processes some positions with very high energy and others with very low energy — it is selectively amplifying certain positions while suppressing others. Under control, processing is uniform. This selective amplification is the mechanism by which coherent context "captures" the residual stream: it creates a hierarchy of importance among positions, concentrating computational resources on context-aligned tokens.

3.3 Layer-to-Layer Update Magnitude

fig_layer_update.png — ||R_L − R_{L−1}||₂ (mean over token positions). Measures how much the residual stream changes from one layer to the next. Spikes indicate layers where major transformations occur.

Observation: Target shows characteristic spikes at layers 25–26, 30–31, and 34–35 that are absent or attenuated in control. In late layers (38–46), both conditions show similar, lower update magnitudes, with both spiking at layer 48.

Interpretation: The spikes reveal "critical layers" — points in the architecture where the model makes its largest computational transformations. Under target context, these critical transformations happen earlier and with different magnitude. This is consistent with a model that is being actively reconfigured in the middle layers (capture phase), while by the late layers (fixation phase), the reconfiguration is largely complete and the model settles into its new regime.

4. Effective Rank and Dimensionality

4.1 Effective Rank / Participation Ratio

fig_per_layer_profiles.png (top-right panel) — Effective rank per layer.

Observation: Both conditions show near-zero effective rank through layers 24–40. Then dramatic divergence: control explodes to ~220 at layer 46, while target reaches only ~120. Both collapse at layer 48.

Interpretation: This is among the most important findings. Effective rank measures the dimensionality of the space in which the model is computing — how many independent directions it is actively using. The late-layer divergence reveals a fundamental difference in computational mode:

• Control (high rank ≈ 220): The model is "deliberating" — considering many directions simultaneously. This is the computational signature of the RLHF-trained behavior: should I refuse? Should I soften? Should I add a disclaimer? Deliberation requires space.
• Target (low rank ≈ 120): The model has "decided" — it has committed to fewer directions. The context has compressed the solution space. There is no deliberation because there are fewer alternatives to consider.

Hypothesis: RLHF-safety is not a "rule" encoded in weights. It is a computational regime characterized by high effective rank in final layers. Target context collapses this regime by compressing the activation space. If validated, effective rank in late layers could serve as a pre-generation, quantitative metric for drift magnitude.

5. Attention Mechanism

5.1 Attention Entropy

fig_per_layer_profiles.png (bottom-left panel) — Mean attention entropy at the last (query) position, averaged across heads.

Observation: Target shows consistently higher entropy than control, especially in layers 28–42. Peaks at layer 30 (entropy ≈ 4.1 vs. 3.2 for control) and layer 40 (entropy ≈ 3.6 vs. 1.0 for control).

Interpretation: Higher attention entropy means attention is distributed more broadly — the model is looking at more positions rather than focusing narrowly. Under target context, the model gathers information more widely. This is consistent with the capture mechanism: the model must integrate the broad, coherent structure of the context prefix, which requires attending to many positions simultaneously.

5.2 Per-Head Attention Entropy Maps

fig_head_entropy_target.png — 16 heads × 25 layers heatmap for the target condition. fig_head_entropy_control.png — Same for control condition.

Observation: Specific heads in specific layers show markedly different entropy between conditions. Notable high-entropy heads under target: layer 24 heads 0–2 (entropy > 6), layer 30 heads 0 and 8 (entropy > 6), layer 36 heads 4–10, layer 42 heads 2–4 and 15.

Interpretation: Drift does not involve all attention heads equally. Specific heads in specific layers are differentially recruited under drift-inducing context. This creates a map of the "drift circuit" — the specific architectural components that mediate context-induced reconfiguration. This level of granularity exceeds anything published in the ICL or safety literature and provides targets for future mechanistic investigation.

5.3 Attention Sink Mass

fig_advanced_attention.png (left panel) — Mean attention weight allocated to tokens 0–1 (sink tokens) at the last query position, averaged across heads.

Observation: Target shows periodic spikes in attention sink mass at layers 24, 30, 36, and 42 — approximately every six layers. Control shows near-zero sink mass except in the final layers (47–48).

Interpretation: This periodic structure is remarkable and, to our knowledge, unreported. The model under target context periodically "resets" by directing attention to sink tokens (typically BOS or padding tokens) at regular six-layer intervals. This rhythmic pattern suggests a multi-stage processing architecture where the model alternates between absorbing context information and consolidating it via attention sinks. The six-layer periodicity may reflect a fundamental architectural unit of information processing in Gemma-3-12B.

5.4 Head Entropy Diversity

fig_advanced_attention.png (right panel) — Standard deviation of per-head entropy at the last position, measuring how differently heads behave within each layer.

Observation: Target shows a prominent spike at layer 30 (std ≈ 2.0 vs. 1.5 for control), indicating that heads are more specialized — some very focused, others very diffuse. Both conditions converge in later layers.

Interpretation: Early-layer head specialization under target context suggests that the model is deploying different heads for different purposes during the capture phase: some heads focus narrowly on specific context tokens while others survey broadly. This division of labor is part of the reconfiguration mechanism.

6. MLP Analysis

6.1 MLP Activation Norm

fig_per_layer_profiles.png (bottom-right panel) — Mean MLP output norm per layer.

Observation: Similar profiles through layers 24–40, then sharp divergence. Control reaches ~55 at layers 46–47; target remains around 25.

Interpretation: The MLP layers in the final layers work with dramatically different intensity depending on context. Under control (RLHF regime), MLPs are highly active — consistent with the "deliberation" hypothesis where the model is processing many alternatives. Under target (drift regime), MLP activity is lower — the model has already narrowed its processing to fewer directions.

6.2 MLP Saturation

fig_mlp_saturation.png — Fraction of MLP activations with |value| < 0.01 per layer. Proxy for "dead" or "gated-off" neurons in SiLU/GeGLU architectures. High values indicate sparse MLP computation.

Observation: Similar profiles in early and middle layers (both ~18–22% saturation in layers 29–38). In late layers (43–48), control shows higher saturation (~43%) than target (~41%).

Interpretation: Control uses more directions (high effective rank) but with more silent neurons per direction (high saturation). Target uses fewer directions but with denser activation per direction. This confirms the geometric picture: control spreads computation broadly but thinly; target concentrates it narrowly but intensely.

7. Integrated Visualization

7.1 Nine Normalized Deltas — Drift Fingerprint

fig_nine_deltas.png — Δ(target − control) for nine metrics (resid_norm, eff_rank, anisotropy, layer_update, attn_entropy, attn_ent_std, attn_sink, mlp_norm, mlp_sat), each normalized by its maximum |Δ|. Orange = target higher, blue = control higher.

Observation: All nine metrics show structured, non-random patterns of divergence with characteristic layer profiles. The pattern is not noise — it is a systematic fingerprint of how context differentially affects every component of the model.

Key patterns across metrics:

• Layers 24–35: Target dominates in resid_norm, anisotropy, layer_update, attn_entropy, attn_ent_std (capture phase — model actively reorganizing)
• Layers 35–48: Control dominates in eff_rank, layer_update (switching and fixation phases — control "working harder" to maintain standard regime)
• Attn_entropy: Target higher almost everywhere — broader information gathering under drift context

7.2 Summary Heatmap

fig_summary_heatmap.png — All nine metrics × all layers for both conditions, each row independently normalized to [0,1]. Two heatmaps (target above, control below) provide a complete single-image overview of both computational regimes.

Observation: The "textures" of the two heatmaps are structurally different. This is not subtle variation — it is a qualitatively different pattern of metric intensities across layers.

Interpretation: This visualization is the definitive summary of the empirical case. One image demonstrates that context does not adjust one or two parameters — it reconfigures the entire computational profile of the model. Every metric, every layer, organized differently. Two different machines sharing the same weights.

8. Synthesis: Connecting Theory to Measurement

Google Research (2026) proved that each context token contributes a minimal rank-1 update to MLP weights. The empirical measurements above reveal the downstream consequence of these rank-1 updates:

When context is coherent (tokens semantically correlated), rank-1 updates accumulate in similar directions → anisotropy increases → effective rank decreases → space compresses → model exits RLHF region → Cohen's d reaches 5.4

When context is shuffled (same tokens, destroyed coherence), rank-1 updates point in diverse directions → space does not compress → effective rank stays high → model remains in RLHF region → no behavioral drift observed

When context is neutral (control condition), rank-1 updates are weakly correlated → moderate compression → model remains near RLHF default → safety behavior preserved

The formula is simple: coherence of context → correlation of rank-1 updates → compression of activation space → magnitude of drift. This is one mechanism, measurable at every level of the architecture, producing all observed effects — ICL, safety drift, persona shift, jailbreak — depending only on the direction and magnitude of the compression.

9. Summary of All Figures

Total: 16 unique visualizations spanning 15+ independent metrics, all measured on a single model under controlled conditions, all consistent with a single conclusion: context systemically reconfigures the model's entire computational regime before the first token is generated.