Published October 21, 2022 | Version v1
Journal article Restricted

A Federated Approach to Predicting Emojis in Hindi Tweets

  • 1. University of Alberta
  • 2. Georgia Institute of Technology
  • 3. Stanford University
  • 4. Santa Clara University
  • 5. Dwarkadas J. Sanghvi College of Engineering
  • 6. Simon Fraser University

Description

This dataset for emoji topic prediction was collected by scraping ~1M tweets. We only kept the 24,794 tweets that are written in Hindi and contain at least one emoji. We duplicated all tweets that contain multiple emojis by the number of emojis contained, assigning a single emoji to each copy, which resulted in the final dataset of 118,030 tweets with 700 unique emojis.

Due to the imbalanced distribution of emojis in our dataset, we assign emojis to 10 coarse-grained categories. This reduction i.e., from multi-label to multi-class and unique emojis into categories, risks losing the semantic meaning of emojis. Our decision is motivated by how challenging emoji prediction is without such reductions.

We pre-processed our data to limit the risk of over-fitting to rare tokens and platform-specific tokens. For instance, we lowercase all text and removed numbers, punctuation, and retweet markers. We replaced mentions, URLs, and hashtags with specific tokens to avoid issues of over-fitting to these.

More information about the dataset is available in our paper.
 

Files

Restricted

The record is publicly accessible, but files are restricted to users with access.

Request access

If you would like to request access to these files, please fill out the form below.

You need to satisfy these conditions in order for this request to be accepted:

1. Data should be used for non-commercial research use.
2. The data should not be shared outside of the research team.
3. The data should not be used for user profiling.
4. Requesters should have received institutional approval (e.g. the requesters have been granted IRB approval for their project).
5. A research statement that details potential harms.

You are currently not logged in. Do you have an account? Log in here