Published June 12, 2026 | Version v4
Preprint Open

Attention Gravity: A System-Level Behavioral Evaluation Framework for Reliable Long-Context LLM Assistants

Authors/Creators

  • 1. Jinan Central Hospital Affiliated to Shandong First Medical University

Description

Intelligent assistants based on long-context large language models (LLMs) face a system-level
reliability problem: they must preserve useful conversational state while obeying user attempts to
switch tasks. This paper studies a recurring failure mode in which a model imports obsolete terms,
entities, constraints, affective framing, or answer formats from an earlier topic into a new one. We call
the behavior contextual inertia and propose Attention Gravity as a bounded behavioral framework
for measuring and mitigating it from input-output traces. The updated evidence package integrates
Qwen, DashScope Qwen, SiliconFlow, and OpenRouter experiments across English, Chinese,
controlled industrial prompts, and source-grounded industrial scenarios. The package contains 39,569
annotated rows, of which 39,567 have valid success/partial/failure labels; the overall heterogeneous
carryover rate is 6.5%, but this pooled number is used only as an audit statistic rather than a primary
claim.
The strongest controlled English Qwen validation shows that 20-turn weak-switch carryover reaches
20.8%, compared with 0.8% at 0 turns. At 20 turns, interventions reduce carryover from 20.8% to 1.8%
when explicit boundary, concrete new-task, and relevance-filter conditions are pooled. The newly
added DashScope Qwen Chinese main validation contributes 8640 responses and shows the same
direction: 20-turn weak-switch carryover is 15.9%, compared with 1.8% at 0 turns. Its Chinese
counterfactual ablation shows original old-topic context at 18.3% versus 6.1% for weak-switch
controls; an explicit-boundary robustness slice reduces original-context carryover to 1.9%, with no
positive original-over-control effect. A completed 2304-row multi-length, two-seed addendum keeps
weak-switch original-context carryover high across 5-, 10-, and 20-turn contexts while showing that
explicit boundaries sharply suppress, though do not entirely eliminate, residual carryover.
Cross-platform comparison strengthens but also bounds the claim. English 20-turn weak-switch
carryover ranges from 2.8% to 20.8% across Qwen, SiliconFlow, and OpenRouter, while Chinese
industrial 20-turn weak-switch carryover remains high across platforms, ranging from 18.3% to 28.9%.
Real-data-grounded industrial prompts show aggregate carryover but low harmful carryover, indicating
that industrial systems require controlled continuity rather than indiscriminate forgetting. A 480-row
stratified human validation confirms that the rubric is human-reproducible (four-class agreement
93.3%, kappa = 0.831; binary carryover kappa = 0.926), while the automatic judge should be treated as
a high-recall, low-precision conservative screen (binary carryover recall 95.8%, precision 47.1%). We
conclude that Attention Gravity is a reproducible behavioral evaluation framework, not a proven
universal mechanism of self-attention.

Files

attention_gravity_acm_tist_manuscript_20260612_resubmission.pdf

Files (47.2 MB)

Name Size Download all
md5:b52dc8e02ac1a366a83d8823865d748b
332.0 kB Preview Download
md5:bf123f9527a44dd9ae0219c21cfc2c28
46.9 MB Preview Download

Additional details

Dates

Updated
2026-06-06
the first version

References

  • [1] A. Vaswani et al., "Attention Is All You Need," in Proc. NeurIPS, 2017.
  • [2] Z. Dai et al., "Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context," in Proc. ACL, 2019.
  • [3] T. B. Brown et al., "Language Models are Few-Shot Learners," in Proc. NeurIPS, 2020.
  • [4] OpenAI, "GPT-4 Technical Report," 2023.
  • [5] H. Touvron et al., "LLaMA: Open and Efficient Foundation Models," 2023.
  • [6] N. F. Liu et al., "Lost in the Middle: How Language Models Use Long Contexts," Transactions of the Association for Computational Linguistics, 2024.
  • [7] G. Xiao et al., "Efficient Streaming Language Models with Attention Sinks," in Proc. ICLR, 2024.
  • [8] S. Jain and B. C. Wallace, "Attention is not Explanation," in Proc. NAACL, 2019.
  • [9] S. Wiegreffe and Y. Pinter, "Attention is not not Explanation," in Proc. EMNLP-IJCNLP, 2019.
  • [10] I. Beltagy, M. E. Peters, and A. Cohan, "Longformer: The Long-Document Transformer," 2020.