Published December 7, 2022 | Version v1
Dataset Open

CommonCrawl News Articles by Political Orientation

  • 1. Maximilian
  • 2. Henning


Dataset description & reproduction steps

The dataset includes news articles gathered from CommonCrawl for media outlets that were selected based on their political orientation. The news articles span publication dates from 2010 to 2021. For more details, please check out our Paper and GitHub repository.

The database file containing the news articles has two main tables, article_urls and article_contents. The tables have the following columns:


  • uuid: An ID that uniquely identifies this URL entry. This column is used as primary key for the table.
  • url: The plain text URL for the news article, as found in CommonCrawl.
  • outlet_name: The name of the news outlet that published the article.


  • uuid: An ID that uniquely identifies this content entry. This column is used as primary key for the table. The key is the same key used in the article_urls table to allow for cross-referencing.
  • date: The automatically extracted publishing date of the article. If it was not possible to automatically extract the date, this field remains empty.
  • content: The plain text content of the article automatically extracted from the crawled HTML document.
  • content_preprocessed: The articles content split by sentences.
  • langauge: The langauge of the article, as identified by the langdetect module (as ISO 639-1 code).


The annotations for political orientations of media outlets were provided by



Files (26.9 GB)

Name Size Download all
32.2 kB Preview Download
16.0 kB Preview Download
4.4 GB Download
9.4 kB Preview Download
22.4 GB Preview Download
26.8 MB Preview Download

Additional details

Related works

Is described by
Peer review: arXiv:2211.03634 (arXiv)