Published December 7, 2022 | Version v1
Dataset Open

CommonCrawl News Articles by Political Orientation

  • 1. Maximilian
  • 2. Henning

Description

Dataset description & reproduction steps

The dataset includes news articles gathered from CommonCrawl for media outlets that were selected based on their political orientation. The news articles span publication dates from 2010 to 2021. For more details, please check out our Paper and GitHub repository.

The database file containing the news articles has two main tables, article_urls and article_contents. The tables have the following columns:

article_urls:

  • uuid: An ID that uniquely identifies this URL entry. This column is used as primary key for the table.
  • url: The plain text URL for the news article, as found in CommonCrawl.
  • outlet_name: The name of the news outlet that published the article.

article_contents:

  • uuid: An ID that uniquely identifies this content entry. This column is used as primary key for the table. The key is the same key used in the article_urls table to allow for cross-referencing.
  • date: The automatically extracted publishing date of the article. If it was not possible to automatically extract the date, this field remains empty.
  • content: The plain text content of the article automatically extracted from the crawled HTML document.
  • content_preprocessed: The articles content split by sentences.
  • langauge: The langauge of the article, as identified by the langdetect module (as ISO 639-1 code).

Notes

The annotations for political orientations of media outlets were provided by allsides.com.

Files

allsides-ranking.csv

Files (26.9 GB)

Name Size Download all
md5:974cea2a8f73f2a53020201e1006cca1
32.2 kB Preview Download
md5:404107c4d8e3882276e5f936c4c1cba0
16.0 kB Preview Download
md5:ef4a29bb699abd6087db09ad5beb7b77
4.4 GB Download
md5:a553d8039739c946198abef7a01483fb
9.4 kB Preview Download
md5:b744d7efabb12346f7ffc7778ff0d3a7
22.4 GB Preview Download
md5:681fe4b303df2d512f4375de27fefd1e
26.8 MB Preview Download

Additional details

Related works

Is described by
Peer review: arXiv:2211.03634 (arXiv)