Aion-BibleQA: Evaluating Retrieval and Citation Faithfulness in Verse-Grounded Bible RAG Systems

This paper introduces Aion-BibleQA, a 40-question pilot benchmark for evaluating retrieval coverage, citation faithfulness, and false-premise robustness in verse-grounded Bible retrieval-augmented generation systems. The work argues that standard retrieval metrics are not enough because a system may retrieve relevant scripture while still using citations incorrectly. Aion-BibleQA separates retrieval quality from citation support and evaluates a layered Bible RAG pipeline using Recall@5, MRR, LLM-judged citation_support, unsupported citation rate, and false-premise refusal rate. The study reports strong pilot results on gold_40 v0.3 and identifies remaining failures as retrieval-scope issues rather than decorative or unsupported citation use.

Preprint version: v1.0