Caveman Compression: MLM-Guided Token Removal for Reducing LLM Inference Costs

Large language models (LLMs) are increasingly deployed in settings where inference cost and latency are dominated by long input contexts. In this work, we propose Caveman Compression, a simple, training-free method for reducing input length while preserving downstream performance. Our approach uses a masked language model (MLM) to estimate token predictability and selectively removes highly predictable tokens, while protecting factual anchors such as named entities, numbers, dates, and compound nouns. This yields compressed prompts that remain semantically faithful but are shorter and cheaper to process. We evaluate Caveman Compression on question answering style benchmarks using multiple LLMs, measuring both answer quality and effective token savings. Across tasks, the method achieves substantial input length reductions (e.g., 30–50% fewer tokens) with only minor drops in accuracy, and in some settings matches the performance of the full input. Because it requires no additional training or model modification, Caveman Compression can be easily integrated into existing LLM pipelines to reduce inference cost and latency.