Datasets from `Discovering and analysing lexical variation in social media text'

This repository contains the datasets that were used in the following three papers, which are also included within P. Shoemark's PhD dissertation `Discovering and analysing lexical variation in social media text':

• P. Shoemark, D. Sur, L. Shrimpton, I. Murray, and S. Goldwater. Aye or naw, whit dae ye hink? Scottish independence and linguistic identity on social media. 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL). 2017. 
• P. Shoemark, J. Kirby, and S. Goldwater. Topic and audience effects on distinctively Scottish vocabulary usage in Twitter data. Workshop on Stylistic Variation at EMNLP. 2017.
• P. Shoemark, J. Kirby, and S. Goldwater. Inducing a lexicon of sociolinguistic variables from code-mixed text. Workshop on Noisy User Generated Text at EMNLP. 2018. 

Datasets consist of tab-separated-values files, in which rows correspond to tweets, with columns for user ID, tweet ID, and timestamp. 

The text of the tweets (and associated metadata) can be re-downloaded (in batches of 100 per request) using Twitter's GET Statuses/Lookup API endpoint (NB: Tweets which have been deleted or made private since the original datasets were collected can not be re-downloaded, so it may not be possible to reconstruct the original datasets in their entirety). 

Most of these datasets were originally drawn from the Sample endpoint of Twitter’s Streaming API (a.k.a. the ‘Spritzer’), which provides a random 1% sample of all public tweets in near real-time:

GU Dataset: Geotagged-UK.zip

Tweets from Sept 2013 - Sept 2014 which are geotagged to locations within the UK.

The file GU_pre-filtering.tsv contains the IDs for all tweets from the ‘Spritzer’ stream which were posted between September 1st 2013 and September 30th 2014, were classified as English by langid.py, are not retweets or quotes, and are geotagged to locations within the UK. •  Tweets: 1,768,334  •  Unique Users: 455,075  • 

The file GU.tsv contains the IDs for tweets in the final GU dataset that was used for the analyses in our EACL 2017 paper, after applying additional pre-processing heuristics to filter out tweets by bots and spammers. •  Tweets: 1,654,204  •  Unique Users: 446,510  • _{(the number of users in the GU dataset was slightly over-counted when reported in the paper; this is the actual number)}

GS Dataset: Geotagged-Scotland.zip 

The subset of Tweets in the GU dataset which are geo-tagged to locations within Scotland, specifically.

The file GS_pre-filtering.tsv contains the IDs for all tweets from the ‘Spritzer’ stream which were posted between September 1st 2013 and September 30th 2014, were classified as English by langid.py, are not retweets or quotes, and are geotagged to locations within Scotland. •  Tweets: 178,401  •  Unique Users: 41,685  • 

The file GS.tsv contains the IDs for tweets in the final GS dataset that was used for the analyses in our EACL 2017 paper, after applying additional pre-processing heuristics to filter out tweets by bots and spammers. •  Tweets: 166,992  •  Unique Users: 40,837  • _{(the number of users in the GS dataset was slightly over-counted when reported in the paper; this is the actual number)}

IT Dataset & Controls: Indyref-Tweets.zip

Tweets from Sept 2013 - Sept 2014 which contain hashtags relating to the 2014 Scottish Independence Referendum (plus 'control' tweets which are by the same users but do not contain referendum-related hashtags)

The file IT_pre-filtering.tsv contains the IDs for all tweets from the ‘Spritzer’ stream which were posted between September 1st 2013 and September 30th 2014, were classified as English by langid.py, are not retweets or quotes, and contain at least one of 47 hashtags we identified as relating to the 2014 Scottish Independence Referendum (see paper for hashtag list). •  Tweets: 77,708  •  Unique Users: 26,019  • 

The file IT.tsv contains the IDs for tweets in the final IT dataset that was used for the analyses in our EACL 2017 paper, after applying additional pre-processing heuristics to filter out tweets by bots and spammers, and tweets which do not contain hashtags that we judged to unambiguously relate to the referendum. •  Tweets: 59,664  •  Unique Users: 18,589  • 

The file IT_controls_pre-filtering.tsv contains the IDs for all tweets from the ‘Spritzer’ stream which were posted between September 1st 2013 and September 30th 2014, were classified as English by langid.py, are not retweets or quotes, and do not contain any of the hashtags we identified as relating to the 2014 Scottish Independence Referendum, but are authored by a user who has also authored a tweet in IT_pre-filtering.tsv. •  Tweets: 1,354,701  •  Unique Users: 26,019  • 

The file IT_controls.tsv contains the IDs for tweets in the final set of Control tweets that was used for the analyses in our EACL 2017 paper, i.e. tweets which do not contain referendum-related hashtags but are authored by users who also have also authored tweets in IT.tsv. •  Tweets: 881,679 •  Unique Users: 18,589  • 

SG-Users’ and IH-Users' Autumn 2014 Timeline Datasets: Autumn-2014_Timelines.zip

Complete tweet histories from Aug-Oct 2014 for users from the GS and IT datasets.

The file SG-Users_Autumn_2014_timelines_pre-filtering.tsv contains the IDs for tweets which were posted in August, September, or October 2014 by users from the GS dataset, i.e. users we know to have used Scottish geotags. This dataset is not restricted to tweets which appear in the ‘Spritzer’ sample; instead it consists of complete User Timelines for the months concerned, retrieved using the statuses/user timeline endpoint of Twitter’s REST API in March 2017. Because there are limits on the number of tweets that can be retrieved using this endpoint, we were not able to retrieve complete Autumn 2014 tweet histories for all of the users in the GS dataset. •  Tweets: 3,014,029  •  Unique Users: 18,274  • 

The file SG-Users_Autumn_2014_timelines.tsv contains the IDs for tweets in the final SG-Users dataset that was used for the analyses in our StyleVar 2017 paper. This dataset consists only of tweets which contain at least one instance of one of 50 lexical variables that were the focus of the study, and has also undergone various other filtering steps; see the paper for full details. •  Tweets: 1,112,931  •  Unique Users: 10,103  • 

The file IH-Users_Autumn_2014_timelines_pre-filtering.tsv contains the IDs for tweets which were posted in August, September, or October 2014 by users from the IT dataset, i.e. users we know to have used Indyref-related hashtags. This dataset was collected in the same manner as SG-Users_Autumn_2014_timelines_pre-filtering.tsv; however, due to an error in this process, the IDs of most of the tweets in this dataset were not recorded. For such tweets the tweet ID column instead contains a placeholder tweet ID of the form _<user_ID>_<month>_<integer>, where the integer denotes the tweet's position in the reverse-chronological list of tweets that were retrieved for that user from that month (e.g. _147527441_09_286  is the placeholder tweet ID we assigned to the 286th September tweet we retrieved from the user whose Account ID is 147527441). Unfortunately, therefore, the tweets in this file whose 'IDs' begin with an underscore cannot be straightforwardly re-downloaded using Twitter's free GET Statuses/Lookup API endpoint; but since their user IDs and timestamps are intact, it would still be possible to retrieve them using the (paid-for) Historical APIs. •  Tweets: 6,997,858  •  Tweets whose IDs were recorded: 288,394  •  Unique Users: 14,645 • 

The file IH-Users_Autumn_2014_timelines.tsv contains the IDs for tweets in the final IH-Users dataset that was used for the analyses in our StyleVar 2017 paper. Like the SG-Users dataset, this dataset consists only of tweets which contain at least one instance of one of 50 lexical variables that were the focus of the study, and has also undergone various other filtering steps; see the paper for full details. As with the pre-filtered version, most of the tweet IDs are unfortunately missing in this dataset. •  Tweets: 2,165,320  •  Tweets whose IDs were recorded: 115,366  •  Unique Users: 10,784 • 

US Geotags: Geotagged-USA.zip

Tweets from June 2013 - July 2016 which are geotagged to locations within the USA.

The file GUSA.tsv contains all tweets from the ‘Spritzer’ sample which were posted between June 30th 2013 to July 1st 2016, are classified as English by langid.py, are not retweets, do not contain urls or embedded media, are not by users with more than 1000 friends or followers, and are geotagged to locations within the USA. This dataset (along with the GU Dataset) was used in our WNUT 2018 paper.   •  Tweets:  8,375,573  •  Unique Users: 1,826,260  •