Published November 11, 2026 | Version 1.0.0

Mixture of Scars (MoS) and SCache O(1): Breaking the Dual Memory Wall of Modern Transformers via Activation-Aware Low-Rank Scar Experts and Widrow-Hoff Associative Memory

Authors/Creators

Description

Autoregressive Large Language Models (LLMs) face a compounding Dual Memory Wall: (1) a static parameter wall O(N) requiring tens to hundreds of gigabytes of VRAM to host frontier weights, and (2) a dynamic context wall O(T) where key-value (KV) caches grow linearly with sequence length T. We introduce two synergistic architectural primitives that simultaneously break both walls without retraining from scratch: Mixture of Scars (MoS), which decomposes dense feed-forward and attention projections into INT4 quantized bases paired with activation-aware SVD (ASVD) low-rank residual "scar" experts and sequential layer-group streaming; and SCache (Scar-Cache), an exact O(1) constant-space dynamic memory combining attention sinks, a compact local KV window, and per-KV-head associative fast-weight slots updated via the orthogonal Widrow-Hoff Delta Rule with ReLU manifold gating.

To prevent benchmark facades, we introduce a 7-vector Adversarial Red-Team Falsification Suite enforcing recursive PyTorch storage-pointer byte audits (0.00 KB discrepancy) and in-flight active causality ablations across real pretrained LLMs (Qwen2.5-0.5B-Instruct 24L, Qwen2.5-72B-Instruct 80L, and Mistral Large 2 123B). Empirically, MoS reduces single-model MLP RAM by 68.6%, cuts 8-specialist multi-tenant VRAM by 92.6% (13.58x), reduces out-of-domain logit KL divergence by 2.45x-4.22x over pure INT4, and streams 72B/123B backbones within < 3.8 GB peak RAM. Simultaneously, SCache bounds dynamic memory to strictly 0.00 marginal bytes beyond window saturation (-91.4% RAM at L=1,280), eliminates Hebbian contradictory ghosting (cosine 1.00000 vs. 0.467), achieves 22.58x higher exact probability on evicted needles than StreamingLLM with zero fine-tuning, and preserves full-stack QA accuracy.

Files

Mixture_of_Scars_and_SCache_Preprint_v1.0.pdf

Files (265.1 kB)

Name Size
md5:d12bf2d8f64a85a884fcb9c893e73d9b
197.0 kB Preview Download
md5:1fa90411d69366ac0226c6d8246d6db1
68.0 kB Preview Download

Additional details

Related works

Is supplemented by
Software: https://github.com/byte-od/mixture-of-scars (URL)