LLM-as-a-Judge: Reassessing the Performance of LLMs in Extractive QA

Extractive reading comprehension question answering (QA) datasets are typically evaluated using Exact Match (EM) and F1-score, but these metrics often fail to fully capture model performance. With the success of large language models (LLMs), they have been employed in various tasks, including serving as judges (LLM-as-a-judge). In this paper, we reassess the performance of QA models using LLM-as-a-judge across four reading comprehension QA datasets. We examine different families of LLMs and various answer types to evaluate the effectiveness of LLM-as-a-judge in these tasks. Our results show th

Research goal: How does the robustness to noisy or irrelevant context in multi-hop HotPotQA questions change when using a large context window (e.g., 128K) versus iterative retrieval with reranking, measured by F1 score and precision under adversarial distractor insertion?

Autonomous synthesis report generated by SOVEREIGN Research Kernel. Tribunal consensus score: 7.8/10.