Accuracy-Matched Is Not Cost-Matched: Pricing INT4 Reasoning Deployments by Served GPU Time per Solved Problem
Description
An INT4 deployment of a reasoning model is a choice of quantizer, repair, export form and decoding control, usually made by accuracy, generated tokens or a fixed-length benchmark, none of which sees how a serving engine charges for length. We serve Qwen3-4B in vLLM on an RTX 4090 and price each configuration by its measured GPU time per solved problem. Under KV-cache-limited load, INT4's extra tokens consume 41–47% of its per-token saving over BF16, against 6% at batch 1, and forcing output lengths and shrinking the KV pool trace this excess to KV-capacity preemption. A repair's time cost changes sign with its serving form: a distillation adapter served unmerged costs more than its INT4 base at batch 1, while merged and requantized it costs less at both loads. Configurations that match BF16's accuracy span a 1.61-fold range of GPU time per solved problem, ParoQuant's own kernel included; pass@1 has no rank agreement with this cost, and a fixed-length benchmark picks a configuration 21.4% costlier than the best. A scheduler replay fitted under load predicts five configurations' relative cost at unprofiled loads on the same GPU within 5.2%, and since on harder workloads and a second model the two configurations cheapest on GSM8K are never both accuracy-matched, each workload is screened and priced on its own.
Other
Version 5 recasts the paper around the GPU time per solved problem measured when each INT4 deployment configuration of a reasoning model (quantizer, repair, export form and decoding control) is served, and the title changes accordingly. It adds served measurements of configurations that match BF16's accuracy, including ParoQuant, requantized forms and a decoding penalty; experiments with forced output lengths and a reduced KV-cache pool that trace the extra cost of longer outputs under load to KV-cache preemption; a scheduler replay that predicts relative cost at loads that were not profiled; results on harder workloads and a second model; and repeated GPTQ quantization runs. It also revises the reading of the merged-and-requantized adapter in version 4: on GSM8K, requantizing the INT4 base without an adapter shortens its outputs by a similar factor, so the requantized form's extra saving over the unmerged adapter comes from the second quantization rather than from the adapter.
This manuscript is a non-peer-reviewed preprint.