Pyramid Cache: Layer-Adaptive KV Cache Compression with Signature-Based Cold Storage

We present Pyramid Cache, a layer-adaptive compression architecture for transformer KV

caches combined with a signature-based cold storage mechanism. Our approach exploits

three properties: (1) token-identity structure varies systematically across layers, enabling

layer-adaptive compression; (2) direction-only signatures predict attention relevance with

0.926 Spearman correlation; and (3) only 0.4% of tokens receive meaningful attention at

any step. A proof-of-concept implementation demonstrates that the system correctly

answers both factual retrieval and complex reasoning questions — including multi-hop

reasoning, causal chains, and contradiction detection — while skipping 99% of tokens

(attending to only 5-8 out of 500+). Complex reasoning tasks show a similarity gap of only

0.0215 compared to simple retrieval, with an EXCELLENT verdict at 1% reconstruction

budget. Our approach is complementary to existing quantization methods like TurboQuant

and represents a fundamentally different strategy: rather than compressing all vectors

equally, we identify and skip the irrelevant ones entirely.