Impact of Multi-Hop QA Benchmark Choice on RAG Retriever Evaluation via F1 Score Analysis

Retrieval-augmented generation (RAG) enhances large language models (LLMs) with external knowledge to answer questions more accurately. However, research on evaluating RAG systems-particularly the retriever component-remains limited, as most existing work focuses on single-context retrieval rather than multi-hop queries, where individual contexts may appear irrelevant in isolation but are essential when combined. In this research, we use the HotPotQA, MuSiQue, and SQuAD datasets to simulate a RAG system and compare three LLM-as-judge evaluation strategies, including our proposed Context-Awar

Research goal: Does the choice of multi-hop QA benchmark (HotPotQA vs. MuSiQue vs. SQuAD) significantly affect the evaluation of RAG retriever strategies, and how can this be measured via F1 score differences across datasets?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 8.4/10.