Monthly Twitter n-grams generated from a corpus of more than 2 billion English tweets (2013-2023)

This dataset contains 1-, 2-, and 3-grams generated from a corpus of more than two billion English tweets from the random 1% sample of the Twitter streaming API harvested between January 2013 and June 2023.

The tweets are a subset of the tweets that were used to generate TweetsKB. Starting from 3,716,933,904 English tweets, 1,606,843,572 retweets and 2,718,439 duplicates (due to redundant harvesting) were removed, resulting in 2,107,371,893 tweets. All URLs and @mentions of user names were removed from their textual content and the text was tokenised using twokenize.

In one way or another, the following people contributed to the creation of this dataset: Erdal Baran, Ernesto Diaz-Aviles, Stefan Dietze, Dimitar Dimitrov, Elso Dittfeld, Pavlos Fafalios, Vasileios Iosifidis, Robert Jäschke, Sebastian Schellhammer, Sebastian Tiesler, Yudong Zhang, Asmelash Teka Hadgu, Ran Yu, Xiaofei Zhu, Matthäus Zloch.

The dataset consists of two parts: one part where the case of letters is preserved and one where all letters are normalised to lower case (file name prefix lc_).  Each part consists of eleven TAR files (one for each year) and each TAR file consists of up to 12 gzip-compressed TSV files (one for each month of the year). Overall, there are 125 TSV files containing n-grams and their monthly frequencies from 01/2013 to 06/2023. Each line in a TSV file represents an n-gram (sorted descending by frequency) and has the following columns:

• ngram_type: 1, 2, or 3 (for 1-grams, 2-grams, and 3-grams, respectively)
• count: the frequency of the n-gram, that is, the number of times it appears in that month (not the number of tweets, since an n-gram can occur several times in one tweet)
• ngram: the n-gram itself

As an example, the first ten rows (plus header) of the file 2018-01.tsv.gz are:

ngram_type      count   ngram
1       195891  .
1       145089  the
1       142295  ,
1       134689  to
1       111336  a
1       106765  I
1       98259   …
1       85359   and
1       77553   you
1       73279   of