Published October 28, 2026 | Version v2

Trust, But Verify: A Claim-Level Citation Audit of AI-Generated Strategic Research Briefs

  • 1. Oryx Centre for Strategic Futures

Description

Version 2 (28 October 2026) replaces version 1 with Study 2: 30 fresh briefs, 194 citations, including a GLM-4.7-Flash replacement arm after the pre-registered Gemini arm failed on rate limits; Study 1 remains the pilot, reported separately and never pooled. AI-generated research briefs read fluently and cite confidently. This paper asks whether their citations survive inspection. We generated ten research briefs on AI governance, strategic foresight and public policy with a single commercial language model under typical user conditions, then audited every cited source: does the URL resolve, and does the resolved page match the cited title? Of 66 citations, 17 (26%) verified, 31 (47%) were broken links, 7 (11%) resolved to a different document than cited, and 11 (17%) could not be verified because the host blocked automated fetching. Separately, only 93 of 190 factual sentences (49%) carried an inline citation at all. And in a claim-support check on all 17 verified citations, only 5 fully supported the brief's claim, 7 partially supported it, and 5 did not support it. The finding is not that models 'lie' — it is that citation fluency is a poor proxy for citation integrity, and that the failure mode is concentrated and detectable. A brief whose citations are not checked is a draft, not a deliverable. All briefs, prompts and the full audit sheet are published alongside this paper so the result can be re-run and disputed.

Other

Contact: amartya@oryxcentre.dpdns.org

Files

Oryx_Paper2_Trust_But_Verify_v2.pdf

Files (103.3 kB)

Name Size
md5:d4de5e34b5078f645c09e7d5e3d50f1c
103.3 kB Preview Download