CPU-Offloaded KV Cache: Enabling Long-Context Inference on Consumer GPUs

We present a simple yet effective method for running large language model inference with long contexts on consumer-grade GPUs with limited VRAM. Our approach offloads the Key-Value (KV) cache from GPU VRAM to CPU RAM after each token generation step, bringing it back to GPU only when needed. We demonstrate that this technique reduces peak VRAM usage by up to 60% on a GTX 960 (4GB VRAM) while maintaining full output quality, enabling contexts of 3500+ tokens on hardware that would otherwise fail with Out-of-Memory errors. The overhead in generation speed is approximately 12-23%, which we argue is an acceptable trade-off for dramatically expanded hardware accessibility. We further validate the approach on Mistral-7B running on a Tesla T4 (14.56GB VRAM), demonstrating that peak VRAM consumption exceeds physical memory limits at 2048+ token contexts — only possible due to active CPU offloading.