Optimizing a Hybrid Retrieval System for Accuracy, Speed, and Cost at Once

This practitioner's report describes how a production retrieval-augmented generation (RAG) system was optimized for accuracy, speed, and memory cost at the same time — a combination usually treated as a set of trade-offs. The system retrieves from a large corpus of long, hierarchically structured documents by fusing a dense (embedding) signal with a lexical (BM25-style) one, then generates cited answers with a large language model.

The central finding is that the largest wins were cheap and structural rather than compute-heavy. One-bit (sign) vector quantization with an exact float rescore shrank the index by roughly 32× and removed the need for a dedicated vector database; an int8 encoder made the model about 4× smaller and 5× faster with byte-identical output; and a handful of lexical and structural ranking priors — title field weighting, home/sub-domain tier routing, and boilerplate demotion — improved accuracy at essentially no query-time cost. Equally important, several "obvious" heavy levers — a cross-encoder reranker, a second round of fine-tuning, and query rewriting — turned out neutral or negative once measured end to end. The report also covers evaluation methodology (semantic LLM judging versus brittle string matching), domain fine-tuning of the embedder, and the safety conditions for a semantic answer cache. All quantities are reported as ratios and relative changes rather than absolute, corpus-specific figures.