Beyond Uniform RRF: Query-Adaptive Weighting for Rank Fusion

Reciprocal Rank Fusion (RRF) is the dominant method for combining multiple retrieval systems, yet standard RRF assigns uniform weights to all retrievers regardless of query characteristics. We propose Adaptive RRF, a lightweight extension that derives per-query retriever weights by scoring a small pilot set of top-ranked documents with a cross-encoder. By scoring as few as 10 documents per retriever (30 total cross-encoder inferences), Adaptive RRF estimates retriever quality for each query and computes weighted fusion scores accordingly. We evaluate on 8 BEIR benchmark datasets spanning 4,176 system-query evaluations. Adaptive RRF significantly outperforms uniform RRF on all 5 discriminative datasets (p < 0.01, paired t-test), achieving a macro-averaged NDCG@10 of 0.6233 versus 0.6032 for uniform RRF—a +3.3% relative improvement. Notably, per-query adaptive weighting surpasses even globally-optimal oracle-weighted RRF (+0.0083 NDCG@10), demonstrating the value of query-level weight adaptation. The pilot mechanism adds negligible compute overhead (30 inferences versus the 100-1000 typical of full reranking) while correctly identifying retriever quality differentials with Pearson correlations of r = 0.27 - 0.54.