fgaim/Tigrinya-PLMs

Monolingual Pre-trained Language Models for Tigrinya # Monolingual Pre-trained Language Models for Tigrinya This repository contains resources for the paper "Monolingual Pre-trained Language Models for Tigrinya" that first appeared at the fifth WiNLP Workshop held in conjunction with EMNLP 2021 on November 11, 2021, in Punta Cana, Dominican Republic. **Abstract** > Pre-trained language models (PLMs) are driving much of the recent progress in natural language processing. However, due to the resource-intensive nature of the models, under-represented languages without sizable curated data have not seen significant progress. Multilingual PLMs have been introduced with the potential to generalize across many languages, but their performance trails compared to their monolingual counterparts and depends on the characteristics of the target language. In the case of the Tigrinya language, recent studies report a sub-optimal performance when applying the current multilingual models. This may be due to its writing system and unique linguistic characteristics, especially when compared to the Indo-European and other typologically distant languages that were used to train the models. In this work, we pre-train three monolingual PLMs for Tigrinya on a newly compiled corpus, and we compare the models with their multilingual counterparts on two downstream tasks, part-of-speech tagging and sentiment analysis, achieving significantly better results and establishing the state-of-the-art. ## Pre-trained Language Models The transformer language models are published on the Huggingface Hub: - TiRoBERTa base, 125M parameters. - TiBERT base, 110M parameters. - TiELECTRA small, 14M parameters. ## Downstream Task Models We fine-tuned the above three models on two tasks. ### Part-of-Speech Tagging Models fine-tuned on the Nagaoka Tigrinya Corpus (NTC) (Tedla et al. 2016). - TiRoBERTa POS - TiELECTRA POS ### Sentiment Analysis Models fine-tuned on a dataset for Tigrinya Sentiment Analysis (Tela et al. 2020). - TiRoBERTa Sentiment - …