DKV: Anchor + Low-Rank Differential KV-Cache Compression for Scalable Long-Context Inference

The key-value (KV) cache is the memory wall of long-context transformer inference: it grows
linearly with sequence length, and on commodity hardware it, not the model weights, is what
first exhausts memory.

DKV is a KV-cache compression runtime that keeps recent tokens exact and compresses older ones.
Each 256-token block is reduced to an anchor token (kept exact), a rank-32 joint K|V truncated-SVD
delta, and a budget of exact residual tokens - the rows the low-rank basis reconstructs worst,
which are precisely the distinctive tokens (digits, names, codes) that verbatim recall depends on.
At decode time a fused buffer routes to the top-K relevant blocks, scores the query in low-rank
space without ever decompressing K, attends residuals and a dense recency window exactly, and
merges the halves with a flash-style log-sum-exp reduction. Prefill uses training-free
block-sparse attention, so its cost grows sub-quadratically.

Implemented entirely in MLX and measured on Qwen2.5-1.5B (int4) on an Apple M3 with 8.6 GB of
unified memory, against two dense full-KV baselines: a memory-optimized mlx_lm engine sharing
DKV's exact int4 weights (the controlled comparison), and a standard PyTorch engine on the
unquantized fp16 checkpoint (a practitioner-default reference, not a weight-matched control).

DKV holds a bounded KV state 1.44x-2.25x smaller per block than a dense cache, recovers a buried
passcode exactly at every context from 4k to 64k, and reaches 64k - where the default PyTorch
full-KV configuration runs out of memory at 16k, and which even the memory-optimized dense cache
reaches only by prefilling 1.72x slower. The cost is per-token decode throughput wherever a dense
cache still fits; this trade-off is reported in full rather than hidden. A CUDA/Triton engine is
implemented and CPU-verified, but no GPU numbers are collected and no GPU performance claims are
made.

All numbers are measured on the described host; none are estimated.