Wikipedia Complete Citation Corpus

Wikipedia Complete Citation Corpus (WCCC) is a corpus of citations, references and sources mined from the English Wikipedia. WCCC was created as a knowledge base used in a machine-learning model for recommending reliable sources to support (or refute) a given textual claim, but can be used for many other purposes.

The solution was described in the paper "Countering Disinformation by Finding Reliable Sources: a Citation-Based Approach", presented at the 2022 International Joint Conference on Neural Networks (IJCNN 2022). Please refer to the article (in conference proceedings or authors' version) for more information on the process of mining the corpus, comparisons with similar resources and the role it plays in recommending sources. The research was done within the HOMADOS project at the Institute of Computer Science, Polish Academy of Sciences.

WCCC contains 4.8 million documents with 50.8 million citations of 24.3 million sources. The dataset is divided into 10 parts (WCCC-part0.zip to WCCC-part9.zip) with approximately the same size. Each of the parts contains batch archives (e.g. batch130.zip), each covering up to 1000 Wikipedia articles. An article is identified by its ID number and described by the following files:

• <ID>_text.txt: the textual content of the article,
• <ID>_citations.txt: the citations occurring in this article, saved as tab-separated values of (1) character offset in the textual content and (2) reference ID,
• <ID>_references.txt: the references cited in the article, saved as tab-separated values of (1) reference ID and (one or many) pairs of (2) source ID and (3) location in the source (e.g. page number),
• <ID>_sources.txt: the sources referenced in the article, saved as tab-separated values of (1) source ID and (2) source description (in wikicode).
• <ID>.txt: human-readable text, created by enriching textual content with the article title and reference IDs.

Additionally, article metadata are included in the meta.tsv file. Each line describes a single article through the following tab-separated fields:

• article ID,
• title of the article,
• Wikipedia ID of the article, which can be used to access the article through URL, i.e. https://en.wikipedia.org/?curid=<WIKI_ID>
• length of the textual content of the article (number of characters),
• number of sources in the article,
• number of references in the article,
• number of citations in the article.

Files metaTrain.tsv and metaTest.tsv contain the same information, but split into training and test set, as used in the work.

Please refer to the paper for an in-depth explanation of the data structure (citations, references, sources, etc.). WCCC was created using Wikipedia dump from 01.02.2021, but you can repeat the mining process using a different dump (or different procedure) by using the published source code. If you intend to apply the corpus in a fact-checking use-case, you might also look at the evaluation datasets we publish separately, one of which is based on WCCC with additional elements (e.g. source identifiers: URL/ISBN/DOI).