Evaluating the Impact of Cross-Encoder Re-Ranking on Retrieval-Augmented Generation Performance
Description
Retrieval-Augmented Generation (RAG) systems ground language model outputs in retrieved evidence, but dense retrieval alone does not guarantee that the most answer-relevant passages reach the generator. This paper presents a controlled evaluation of cross-encoder re-ranking in a dense-retrieval RAG pipeline. We implement two systems that are identical except for the presence of a BERT-based cross-encoder re-ranker inserted between the retriever and generator. Evaluating on 200 Natural Questions validation examples over a corpus of 75,000 Wikipedia passages, we find that re-ranking improves Exact Match from 0.075 to 0.095 (+26.7%) and Token F1 from 0.029 to 0.040 (+39.3%). Absolute scores reflect the constrained experimental setting (a 75,000-passage corpus and a 3B-parameter generator), and the relative improvements are the primary contribution. The latency cost is approximately 4.3 seconds per query on CPU hardware, reducible to under 200ms with GPU deployment. To our knowledge, no prior work has isolated the causal effect of inserting a frozen, off-the-shelf cross-encoder re-ranker into an otherwise unchanged RAG pipeline, keeping retriever, generator, prompt, and corpus fixed, while reporting both retrieval and generation outcomes in the same experiment. While the improvement did not reach statistical significance at this sample size (Wilcoxon signed-rank test, p=0.317), the consistent directional effect across both metrics suggests re-ranking may meaningfully benefit RAG answer quality, warranting replication at larger scale.