Webis
Published July 5, 2018 | Version v2
Dataset Open

Webis Wikipedia Text Reuse Corpus 2018 (Webis-Wikipedia-Text-Reuse-18)

Creators

  • 1. Universität Paderborn
  • 2. Bauhaus-Universität Weimar
  • 3. Martin-Luther-Universität Halle-Wittenberg
  • 4. Universität Leipzig

Description

The Wikipedia Text Reuse Corpus 2018 (Webis-Wikipedia-Text-Reuse-18) containing text reuse cases extracted from within Wikipedia and in between Wikipedia and a sample of the Common Crawl.

The corpus has following structure:

  • wikipedia.jsonl.bz2: Each line, representing a Wikipedia article, contains a json array of article_id, article_title, and article_body
  • within-wikipedia-tr-01.jsonl.bz2: Each line, representing a text reuse case, contains a json array of s_id (source article id), t_id (target article id), s_text (source text), t_text (target text)
  • within-wikipedia-tr-02.jsonl.bz2: Each line, representing a text reuse case, contains a json array of s_id (source article id), t_id (target article id), s_text (source text), t_text (target text)
  • preprocessed-web-sample.jsonl.xz: Each line, representing a web page, contains a json object of d_id, d_url, and content
  • without-wikipedia-tr.jsonl.bz2: Each line, representing a text reuse case, contains a json array of s_id (Wikipedia article id), d_id (web page id), s_text (article text), d_content (web page content)

The datasets were extracted in the work by Alshomary et al. 2018 that aimed to study the text reuse phenomena related to Wikipedia at scale. A pipeline for large scale text reuse extraction was developed and used on Wikipedia and the CommonCrawl.

Files

Files (43.6 GB)

Name Size Download all
md5:8dad02cc0553a4c5cc9c6258facbe7da
34.9 GB Download
md5:a063f2d254ca2aef4bef5d56b1959824
3.0 GB Download
md5:8c52b49822a82ff5bd4bc7a58a595a05
3.0 GB Download
md5:0d1a465382385869767b59a7d6e2631e
2.5 GB Download
md5:3ae6cb91b60f5666e81f6bf05ae9e955
190.5 MB Download

Additional details

References

  • Milad Alshomary, Michael Völske, Tristan Licht, Henning Wachsmuth, Benno Stein, Matthias Hagen, and Martin Potthast. Wikipedia Text Reuse: Within and Without. In Leif Azzopardi et al, editors, Advances in Information Retrieval. 41st European Conference on IR Research (ECIR 2019) volume 11437 of Lecture Notes in Computer Science, pages 747-754, Berlin Heidelberg New York, April 2019. Springer.