castorini/afriberta

AfriBERTa: Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages # AfriBERTa: Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages This repository contains the code for the paper ***Small Data? No Problem! Exploring the Viability of Pretrained Multilingual Language Models for Low-resourced Languages*** which appears in the first workshop on Multilingual Representation Learning at EMNLP 2021. AfriBERTa was trained on 11 languages - Afaan Oromoo (also called Oromo), Amharic, Gahuza (a mixed language containing Kinyarwanda and Kirundi), Hausa, Igbo, Nigerian Pidgin, Somali, Swahili, Tigrinya and Yorùbá. AfriBERTa was evaluated on NER and text classification spanning 10 languages (some of which it was not pretrained on). It outperformed mBERT and XLM-R on several languages and is very competitive overall. ## Pretrained Models and Dataset **Models:** We release the following pretrained models: - AfriBERTa Small (97M params) - AfriBERTa Base (111M params) - AfriBERTa Large (126M params) **Dataset**: castorini/afriberta-corpus ## Reproducing Experiments ### Datasets and Tokenizer Below are details on how to obtain the datasets and trained sentencepiece tokenizer: **Language Modelling**: The data for language modelling can be downloaded from this URL **NER**: To obtain the NER dataset, please download it from this repository **Text Classification**: To obtain the topic classification dataset, please download it from this repository **Tokenizer**: The trained sentencepiece tokenizer can be downloaded from this URL ### Training To train AfriBERTa and evaluate on both downstream tasks, simply install all requirements in ```requirements.txt```, download the relevant datasets and run the following script: ``` bash run_all.sh ``` This script will: 1. Train the multilingual language model from scratch and save the model as well as relevant logs 2. Evaluate the trained language model on NER for all ten languages over 5 seeds 3. Evaluate the trained l …