There is a newer version of the record available.

Published July 2, 2026 | Version v1

Verify-RAG: A Human-in-the-Loop Framework for Citation and Groundedness Verification of Retrieval-Augmented Generation

Authors/Creators

Description

Retrieval-Augmented Generation (RAG) grounds large language model (LLM) outputs in retrieved sources, yet hallucinations persist: recent audits find that a large fraction of cited-source responses are not fully supported by the sources they invoke. Automated hallucination detection has matured rapidly, but the design of human-in-the-loop (HITL) workflows for post-generation verification of RAG outputs remains underexplored. Existing HITL surveys concentrate on training-time and data-time human involvement (active learning, RLHF, annotation) and give little attention to the output-verification layer. We present Verify-RAG, a framework that integrates automated hallucination detection with structured human verification at three intervention points: retrieval validation, claim-level groundedness checking, and citation alignment verification. We implement and evaluate the framework's automated detection layer on the RAGTruth benchmark across all three tasks. A fine-tuned span detector reaches a pooled example-level F1 of 0.79 (matching its published benchmark performance), and substantially outperforms a prompt-based GPT-4o-mini judge on the question-answering subset (F1 0.70 vs. 0.44). Yet with a pooled precision of 0.81 and recall of 0.78, roughly one in five flags is a false alarm and about one in five hallucinations is missed — a residual-error profile that motivates human verification rather than full automation in accountability-sensitive deployments. We contribute (i) an analysis of the HITL-literature gap for post-generation RAG verification, (ii) a taxonomy of three intervention points, (iii) an open, reproducible automated-evaluation harness, and (iv) a pre-registered human-study protocol for measuring reviewer effort and appropriate reliance. The framework, evaluation harness, and protocol are released as open-source resources, including a live interactive demo at https://huggingface.co/spaces/Harshita0803/verify-rag

Files

RAG_Verification.pdf

Files (214.2 kB)

Name Size
md5:d9fd4ac08f913b6f086704c866c2f199
214.2 kB Preview Download

Additional details