Quantization as Cache Amplification: Trillion-Parameter Mixture-of-Experts Inference on a Commodity Laptop

Weight quantization is usually justified as footprint reduction. This paper argues that for offloaded Mixture-of-Experts (MoE) inference that framing misses where the leverage is.

In an offloaded MoE engine the binding resource is not storage capacity but the fraction of expert slots resident in DRAM, and storage traffic depends on that fraction through a cache hit rate that is both concave and, for recency-based policies, discontinuous. We show on real routing traces that a least-recently-used expert cache collapses to a hit rate of exactly zero whenever its capacity falls below the k·L expert slots a single token touches — a cyclic-reference pathology measured at precisely the predicted threshold. Quantization is what moves a system across that threshold, so bits buy throughput super-proportionally to their compression ratio.

We make this quantitative on one commodity laptop (NVIDIA RTX A500, 4 GB VRAM; 32 GB DRAM; consumer NVMe). Contributions: a sub-2-bit post-training codec combining randomized Hadamard incoherence processing, multi-stage residual vector quantization and block-LDL error feedback; frequency-conditioned bit allocation, which spends bits on experts in proportion to measured activation frequency and improves perplexity by 13.8% over uniform allocation at identical average rate; and an end-to-end roofline in which every hardware term is measured on the host and the cache term is validated against 49,152 tokens of real OLMoE-1B-7B routing traces.

Measured: 6.01 GB/s unbuffered NVMe random read at expert-block granularity, 6.11 GB/s over PCIe, 88.2 GB/s GPU device bandwidth. The codec reaches WikiText-2 perplexity 12.17 at 2.01 bits/weight against 8.11 in bfloat16. Composing these, a 1,045.8B-parameter MoE occupies 196 GB at 1.5 bits and projects 1.81 tokens/s at batch 1 and 3.08 tokens/s at batch 32: a 16.4x throughput gain from a 10.7x compression. We also report a negative result constraining any such system: sustaining the storage stream while materializing fp16 weights would need ~116 GB/s of device bandwidth against 88.2 GB/s measured, so dequantization must be fused into the GEMM.

Scope: No trillion-parameter model was executed. The 1T figures are an analytical projection from measured host parameters and a cache model validated at 7B scale; full sensitivity across the hit-rate range is reported in the paper.

Code, raw routing traces and all measurement artefacts: https://huggingface.co/datasets/mkvn/quantization-cache-amplification