HP-BERT: A framework for longitudinal study of Hinduphobia on social media via LLMs

1. Introduction

HP-BERT is a specialized variant of BERT designed to detect Hinduphobic content on Twitter. It is fine-tuned using the Hinduphobic COVID-19 X (Twitter) Dataset, which comprises over 8,000 tweets collected during the COVID-19 pandemic (November 2019 – December 2022). The dataset includes 2,000 manually labeled tweets, with additional annotations generated using OpenAI's GPT-3.5 Turbo API.

HP-BERT is further enhanced using the SenWave dataset to improve its sentiment analysis capabilities. It is optimized for detecting Hinduphobia, analyzing sentiment polarity, and providing nuanced insights into the emotional tone of discussions. The model is particularly effective for Hinglish (Hindi-English) text, making it well-suited for Indian social media analysis.

2. Technical Setup

2.1 Model Architecture & Training Configuration

HP-BERT is based on BERT (Bidirectional Encoder Representations from Transformers) and fine-tuned with a multi-stage approach. The model is trained using the following hyperparameters:

• Dropout Regularization Rate: 0.2
• Epochs: 10
• Learning Rate: 2 × 10⁻⁵
• Batch Size: 8
• Weight Decay Regularization Rate: 0.01
• Hardware:
  • NVIDIA GeForce RTX 2080 Ti (11264 MiB memory)
  • AMD Ryzen 7 5800H Processor
  • 16 GB System Memory
• Software Libraries:
  • Transformers
  • NumPy
  • Pandas
  • Matplotlib
  • PyTorch

3. Fine-Tuning & Evaluation

3.1 Hinduphobia Detection

The HP-BERT model was fine-tuned using the Hinduphobic COVID-19 X (Twitter) dataset, splitting it into four different training/testing configurations. The model's performance in classifying Hinduphobic tweets was evaluated using 30 independent experimental runs. The results demonstrate strong accuracy and balanced F1 scores, indicating the model’s ability to maintain both precision and recall.

3.2 Sentiment Analysis with SenWave Dataset

The model was further fine-tuned on the SenWave dataset for sentiment analysis. Since this is a multi-label classification task, conventional evaluation metrics are not applicable. Instead, the following metrics were used:

These results validate the model's effectiveness in multi-label sentiment classification.

4. Applications

HP-BERT has been tested on multiple datasets, including the Global COVID-19 Twitter dataset, covering six countries (Australia, Brazil, India, Indonesia, Japan, and the UK). Its applications include:

• Hinduphobia Detection: Identifying toxic and abusive language.
• Sentiment Analysis: Understanding polarity and emotional tone in social media discourse.
• User Behavior Analysis: Studying how Hinduphobic narratives propagate online.
• Hate Speech Detection: Assisting research on online hate speech dynamics.

5. Availability

HP-BERT is available for public use to facilitate further research in sentiment analysis, hate speech detection, and computational social science.

References

1. Original BERT Model Documentation: https://huggingface.co/docs/transformers/en/model_doc/bert
2. Hinduphobic COVID-19 X (Twitter) Dataset: https://www.kaggle.com/datasets/ashutoshsingh22102/hinduphobic-covid-19-x-twitter-dataset-india
3. SenWave Dataset: https://github.com/gitdevqiang/SenWave