LEX-FVM: A Provenance-Audited Benchmark for Faithfulness Verification in Legal Question Answering

This record contains the provenance-audited benchmark used to evaluate LEX-FVM, a typed, risk-controlled faithfulness verification framework for citation-aware legal question answering over Indian Supreme Court judgments.

benchmark_153.csv — 153 audited question–answer instances. Each row holds the question, reference answer, generated answer, emitted citations, the retrieved evidence pool presented to the generator, a single-annotator faithfulness label with the annotator's note, and a provenance flag. Label composition: 121 fully supported, 26 partially supported, 5 contradicted, 1 unsupported, 0 misleading; 32 instances (20.9%) are not fully supported.

top_k_hits is the evidence column used throughout — the per-instance pool actually shown to the generator, mean 4.93 passages. retrieved_evidence is a different and larger evidence set retained for provenance only; computing against it produces an invalid result set.

excluded_59.csv — every instance removed during the provenance audit, with its identifier and reason. Of 212 annotated instances, 70 answers were byte-identical to the generation log and 61 more matched after encoding and punctuation normalisation, giving 131 provenance-verified; 22 further instances were provenance-untestable and retained, yielding the 153-instance analysis set. Of the 59 excluded, 34 had no answer recorded in the generation log and 25 had a logged answer that could not be reconciled with the annotated text.

results/ — the output tables backing every reported figure.

Annotation was performed by a single annotator in a single round. No second annotator participated, no adjudication was carried out, and no inter-annotator agreement statistic is reported or can be derived from these files.

Derived from the IndicLegalQA dataset (Veningston & Mishra, Mendeley Data, https://doi.org/10.17632/gf8n8cnmvc.2) and the Indian Supreme Court judgments dataset (https://www.kaggle.com/datasets/vangap/indian-supreme-court-judgments). Judgments of the Supreme Court of India are public records and are not subject to copyright under section 52(1)(q) of the Copyright Act, 1957.
Code: [GitHub URL]