Zabanshenas - Language Detector

Zabanshenas A Transformer-based solution for identifying the most likely language of a written document/text. Zabanshenas is a Persian word that has two meanings: A person who studies linguistics. A way to identify the type of written language. Introduction In this repository, I will use another perspective in creating a language detection model using Transformers. Nowadays, Transformers have played a massive role in Natural Language Processing fields. In short, Transformers uses an attention mechanism to boost the speed and extract a high level of information (abstraction). There are plenty of ways, solutions, and packages to find the language of a written piece of text or document. All of them have their pros and cons. Some able to detect faster and support as many languages as possible. However, in this case, I intend to use Transformers to understand similar groups of languages and cover 235 languages thanks to WiLI-2018 and the Transformer architecture. This model can detect a written language in three different stages: paragraph, sentence, and subset of text between three and four tokens. WilI-2018 (Cleaned version) It is a benchmark for monolingual written natural language identification (high covering of a specific language). It contains 229,095 paragraphs that cover 235 languages. Language distribution includes 122 Indo-European languages, 22 Austronesian languages, 17 Turkic languages, 14 Uralic languages, 11 Niger-Congo languages, 10 Sino-Tibetan languages, 9 Afro-Asiatic languages, 6 constructed languages, and 24 languages of smaller families. It also consists of groups of similar languages and dialects: Arabic, Egyptian Arabic English, Old English, Scots Standard Chinese, Min Nan Chinese, Hakka Chinese, Literary Chinese, Wu Chinese German, Bavarian, Low German, Palatine German, Ripuarisch, Alemannic German, Pennsylvania German Belarusian, Belarusian (Taraschkewiza) Kurdish, Central Kurdish Indonesian, Minangkabau, Banyumasan, Banjar, Sundanese, Javanese Languages are spoken in India: Maithili, Bhojpuri Bengali, Bishnupriya Konkani, Marathi Russian, Komi-Permyak Persian, Gilaki, Mazanderani On the other hand, this dataset addresses low-resources languages, as shown in Fig 1: Chechen Haitian Creole Newari Pampanga Model (Architecture) RoBERTa as a candidate model is used for this specific task with the following procedure. Firstly, the model is trained on the WILI-2018 corpus and then fine-tuned as a SequenceClassification task to detect independent and similar languages and dialects. The data is divided into three segments, 70% of the whole data (train + test) as paragraph choices, 15% tokenized into sentences, and what remains, split into the subset of three or five tokens per record to achieve better results. In total, the dataset consists of a 1M training set, 11K validation set, and 11K testing set.