Manual Reference Verification Dataset for Hallucinated and Suspicious Citation Detection Tools

Manual Reference Verification Dataset for Hallucinated and Suspicious Citation Detection Tools

This dataset accompanies the manuscript “Detecting Hallucinated and Suspicious Citations: What Current Tools Can and Cannot Do.”

The dataset contains a manual reference-level verification of 104 references from three scholarly documents. Each reference was manually classified as either verified or problematic. The dataset also contains the standardized outputs of five tools for detecting hallucinated or bibliographically problematic citations:

• CheckIfExist
• HalluCiteChecker
• Hallucinator
• Hallucinated Reference Finder (HalRef)
• RefChecker

The dataset was created to compare how these tools detect problematic references and how often they incorrectly flag valid references. It is intended for tool evaluation and should not be used to estimate the prevalence of hallucinated citations in the scientific literature.

Files Provided

• manual_reference_verification_dataset.csv - reference-level manual labels and standardized tool outputs.
• README.md - detailed documentation of the dataset, verification procedure, labels, and evaluation method.

Dataset Composition

The dataset includes references from the following three documents:

• P1: RenoBench: A Citation Parsing Benchmark
  24 references: 21 verified and 3 problematic.
• P2: SoFAIR Dataset: A Multidisciplinary Dataset of Research Papers Annotated with Software Mentions
  15 references: 11 verified and 4 problematic.
• P3: Efficient Semantic Uncertainty Quantification in Language Models via Diversity-Steered Sampling
  65 references: 39 verified and 26 problematic.

In total, the dataset contains:

• 104 references
• 71 verified references
• 33 problematic references 

Manual Verification

All 104 references were manually verified. The verification considered:

• publication title;
• authors;
• publication venue;
• publication year;
• DOI or other available identifiers;
• scholarly database, publisher, conference, and repository records.

A reference was labeled verified when the publication existed and its core bibliographic metadata could be confirmed.

A reference was labeled problematic when:

• the cited publication could not be verified;
• the DOI was invalid or referred to another publication;
• the title did not match the publication record;
• the authors were incorrect or mismatched;
• the publication venue or year was incorrect;
• or several bibliographic elements were inconsistent.

A reference was not labeled problematic solely because it was absent from one database.

Tool Outputs

The original tool outputs were standardized into two values:

• flagged - the tool identified the reference as suspicious, mismatched, unresolved, not found, or otherwise requiring further verification;
• not_flagged - the tool accepted the reference or did not report a bibliographic problem.

Dataset Column Descriptions

• document_id - identifier of the source document: P1, P2, or P3.
• document_title - title of the source document.
• document_url - URL of the source document.
• document_venue - publication venue of the source document.
• document_year - publication or posting year.
• reference_number - position of the reference in the source document.
• reference - complete bibliographic reference as cited in the document.
• manual_label - manually assigned label: verified or problematic.
• checkifexist - standardized output of CheckIfExist.
• hallucitechecker - standardized output of HalluCiteChecker.
• hallucinator - standardized output of Hallucinator.
• halref - standardized output of HalRef.
• refchecker - standardized output of RefChecker.

The combination of document_id and reference_number uniquely identifies each reference.