Dataset Open Access

PTNews Corpus

Nunes, Davide

The PTNews Corpus is a collection of over 19 million tokens extracted from 10 years of political news articles (in Portuguese) from the Portuguese newspaper PÚBLICO. The corpus is available under the Creative Commons Attribution-NonCommercial-ShareAlike Licence. The material contained on the PTNews Corpus is © 2010-2020 PÚBLICO Comunicação Social SA.

The corpus sizes between the preprocessed version of Penn Treebank (PTB) and WikiText-103. Similarly to WikiText, PTNews has a larger vocabulary than PTB and retains the original case, punctuation and numbers. This corpus contains over 31000 publicly available full articles which makes it well suited for models that can take advantage of long-term dependencies.

The corpus is available as a word-level collection of articles in two version: the first (ptnews_origin) contains a single file with all the articles in the form: title, URL, date, body; the second, contains only the title and body of the news articles and it is split into train, test, validation sets. In this processed version, the words with less than 3 occurrences are mapped to the <unk> token. Each sentence in an article body occupies a single line of the dataset and the end of paragraph is marked with the <eop> tag at the end of a sentence. Portuguese words resulting from contractions like "desta", ou "nesta" are separated into "d", "esta", "n", "esta", respectively.
 

Sample article:
 

Carlos César : Cavaco " cansado e sem entusiasmo " quis afastar responsabilidades sobre a crise
https://publico.pt/2010/06/10/politica/noticia/carlos-cesar-cavaco-cansado-e-sem-entusiasmo-quis-afastar-responsabilidades-sobre-a-crise-1441369
2010-06-10 15:38:00

O presidente do Governo Regional dos Açores , Carlos César , considerou hoje que Cavaco Silva esteve " cansado e sem entusiasmo " no discurso do Dia de Portugal , onde afastou responsabilidades sobre a actual crise . <eop>
" O país ouviu um Presidente cansado e sem entusiasmo , que andou às voltas com os papéis para dizer que não tinha nada a ver com as razões da crise " , afirmou Carlos César , num comentário à Lusa sobre o discurso do Presidente da República na cerimónia oficial do 10 de Junho , realizada em Faro . <eop>
Carlos César considerou , no entanto , " positivo " que Cavaco Silva tenha feito " um discurso alinhado com um tema recorrente na apreciação do momento que vivemos , o da coesão e da corresponsabilização " . <eop>
No mesmo sentido , manifestou concordância com o apelo que Cavaco Silva fez " à responsabilidade dos empregadores e empregados " , mas deixou um alerta relativamente à referência do Presidente da República à necessidade de " limpar Portugal " . <eop>
Para Carlos César , se essa referência " for despida de conteúdo institucional útil , tratou-se de mais um discurso que se perderá na babugem política d aquilo que Cavaco Silva entendeu recordar como o ' rectângulo ' " . <eop>

Reporting Results
If you wish to report results or other resources obtained on the PTNews contact Davide Nunes with the following information:

  • Task: e.g. Language Modelling, Semantic Similarity, etc;
  • Publication URL: url to published article or preprint;
  • Type of Model: LSTM Neural Network, n-grams, GloVe vectors, etc;
  • Evaluation Metrics: e.g. validation and testing perplexities in the case of language modelling.

They will be displayed here

Preprocessed Corpus Statistics

  • articles: 31.919
  • articles by split:
    • train: 25.537
    • test: 3.191
    • val: 3.191
  • unique tokens: 68.318
  • unique OoV Tokens: 76.157
  • total tokens: 19.021.661
  • total OoV tokens: 95.043
  • OoV rate: 0.5%
  • tokens by split:
    • train: 15.242.995
    • test: 1.895.184
    • val: 1.883.482

 

Contact Information

If you have questions about the corpus or want to report benchmark results, contact Davide Nunes.


 

Files (72.8 MB)
Name Size
LICENCE.txt
md5:27671fca8c18e8ee61aad115bb041d9e
21.0 kB Download
ptnews.tar.gz
md5:a6feb6bbba29913daee1e855793bab1f
35.9 MB Download
ptnews_origin.tar.gz
md5:a2082d999ec5394f8de3edc866ba7670
36.9 MB Download
90
109
views
downloads
All versions This version
Views 9090
Downloads 109109
Data volume 3.8 GB3.8 GB
Unique views 7171
Unique downloads 3636

Share

Cite as