Roman Urdu Sentiment Analysis
Description
This report studies four-class sentiment and abuse classification of Roman Urdu text (Negative, Positive, Neutral, Abusive). Roman Urdu is Urdu written in the Latin script, usually mixed with English and spelled phonetically with no fixed standard, which makes it difficult for models built for standardized languages.
A public Roman Urdu sentiment dataset and the RUHSOLD hate-speech dataset were merged under a unified label scheme. The Abusive class was heavily underrepresented in training (3,358 samples against more than 20,000 per other class), so dictionary-based synonym replacement was used to expand it to 13,374 samples. Augmentation was applied to the training split only.
Three models were compared: an LSTM trained from scratch, fine-tuned multilingual BERT (mBERT), and fine-tuned GPT-2. On the unaugmented test set, GPT-2 performed best with 70.24% accuracy and a macro F1 of 0.65, followed by mBERT (65.05%, 0.62) and the LSTM (57.43%, 0.54). Abusive remained the hardest class for every model (F1 0.31–0.42). The results show that pre-trained transformers transfer usefully to Roman Urdu, and that an English-only model can outperform a multilingual one on Latin-script, code-mixed text.
This is a course project report for Natural Language Processing at Habib University, Karachi, Pakistan. It has not been peer-reviewed. Code is available at https://github.com/aqib420/RomanUrduSentimentAnalysis.
Files
Report_NLP.pdf
Additional details
Software
- Repository URL
- https://github.com/aqib420/RomanUrduSentimentAnalysis
- Programming language
- Python
References
- J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of deep bidirectional transformers for language understanding," in Proc. NAACL-HLT, 2019.
- A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, "Language models are unsupervised multitask learners," OpenAI, Tech. Rep., 2019.
- A. Conneau et al., "Unsupervised cross-lingual representation learning at scale," in Proc. ACL, 2020.
- Z. Sharf and S. U. Rahman, "Performing natural language processing on Roman Urdu datasets," Int. J. Comput. Sci. Netw. Secur., vol. 18, no. 1, pp. 141–148, 2018.
- H. Rizwan, M. H. Shakeel, and A. Karim, "Hate-speech and offensive language detection in Roman Urdu," in Proc. EMNLP, 2020.
- J. Wei and K. Zou, "EDA: Easy data augmentation techniques for boosting performance on text classification tasks," in Proc. EMNLP-IJCNLP, 2019. HowMannyMore, "Roman Urdu sentiment dataset," Hugging Face Hub. [Online]:Available: https://huggingface.co/datasets/HowMannyMore/romanurdu-sentiment-dataset
- S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
- T. Wolf et al., "Transformers: State-of-the-art natural language processing," in Proc. EMNLP: Syst. Demonstrations, 2020.