UBC-NLP/serengeti

SERENGETI: Massively Multilingual Language Models for Africa Multilingual pretrained language models (mPLMs) acquire valuable, generalizable linguistic information during pretraining and have advanced the state of the art on task-specific finetuning. To date, only ~31 out of 2,000 African languages are covered in existing language models. We ameliorate this limitation by developing SERENGETI , a set of massively multilingual language model that covers 517 African languages and language varieties. We evaluate our novel models on eight natural language understanding tasks across 20 datasets, comparing to 4 mPLMs that cover 4-23 African languages. SERENGETI outperforms other models on 11 datasets across eights tasks, achieving 82.27 average F 1 -score. We also perform analyses of errors from our models, which allows us to investigate the influence of language genealogy and linguistic similarity when the models are applied under zero-shot settings. We will publicly release our models for research. ## Table of Contents - 1 Our Language Models - 1.1 Training Data - 1.2 Models Architecture - 1.3 Serengeti Models - 2. AfroNLU Benchmark and Evaluation - 2.1 Named Entity Recognition - 2.2 Phrase Chuncking - 2.3 Part of Speech Tagging - 2.4 News Classification - 2.5 Sentiment Analysis - 2.6 Topic Classification - 2.7 Question Answering - 2.8 Language Identification - 3. How to use Serengeti model - 4. Ethics - 5. Support Languages - 6. Citation - 7. Acknowledgments ## 1. Our Language Models ## 1.1 Training Data * **Serengeti Training Data**: SERENGETI is pretrained using 42GB of data comprising a multi-domain, multi-script collection. The multi-domain dataset comprises texts from religious, news, government documents, health documents, and existing corpora written in five scripts from the set {Arabic, Coptic, Ethiopic, Latin, and Vai}. * Religious Domain. Our religious data is taken from online Bibles, Qurans, and data crawled from the Jehovah’s witness website. We also include religious texts from the book of Mormon. * News Dom …