Zabanshenas - Language Detector

Zabanshenas

A Transformer-based solution for identifying the most likely language of a written document/text. Zabanshenas is a Persian word that has two meanings:

• A person who studies linguistics.
• A way to identify the type of written language.

Introduction

In this repository, I will use another perspective in creating a language detection model using Transformers. Nowadays, Transformers have played a massive role in Natural Language Processing fields. In short, Transformers uses an attention mechanism to boost the speed and extract a high level of information (abstraction).

There are plenty of ways, solutions, and packages to find the language of a written piece of text or document. All of them have their pros and cons. Some able to detect faster and support as many languages as possible. However, in this case, I intend to use Transformers to understand similar groups of languages and cover 235 languages thanks to WiLI-2018 and the Transformer architecture.

This model can detect a written language in three different stages: paragraph, sentence, and subset of text between three and four tokens.

WilI-2018 (Cleaned version)

It is a benchmark for monolingual written natural language identification (high covering of a specific language). It contains 229,095 paragraphs that cover 235 languages. Language distribution includes 122 Indo-European languages, 22 Austronesian languages, 17 Turkic languages, 14 Uralic languages, 11 Niger-Congo languages, 10 Sino-Tibetan languages, 9 Afro-Asiatic languages, 6 constructed languages, and 24 languages of smaller families. It also consists of groups of similar languages and dialects:

• Arabic, Egyptian Arabic
• English, Old English, Scots
• Standard Chinese, Min Nan Chinese, Hakka Chinese, Literary Chinese, Wu Chinese
• German, Bavarian, Low German, Palatine German, Ripuarisch, Alemannic German, Pennsylvania German
• Belarusian, Belarusian (Taraschkewiza)
• Kurdish, Central Kurdish
• Indonesian, Minangkabau, Banyumasan, Banjar, Sundanese, Javanese
• Languages are spoken in India:
  • Maithili, Bhojpuri
  • Bengali, Bishnupriya
  • Konkani, Marathi
• Russian, Komi-Permyak
• Persian, Gilaki, Mazanderani

On the other hand, this dataset addresses low-resources languages, as shown in Fig 1:

• Chechen
• Haitian Creole
• Newari
• Pampanga

Model (Architecture)

RoBERTa as a candidate model is used for this specific task with the following procedure. Firstly, the model is trained on the WILI-2018 corpus and then fine-tuned as a SequenceClassification task to detect independent and similar languages and dialects. The data is divided into three segments, 70% of the whole data (train + test) as paragraph choices, 15% tokenized into sentences, and what remains, split into the subset of three or five tokens per record to achieve better results.

In total, the dataset consists of a 1M training set, 11K validation set, and 11K testing set.