AlexisMatzopoulos/elc-bert-isiXhosa

Investigating ELC-BERT for resource-efficient isiXhosa Language Modeling Alexis Matzopoulos mtzale001@myuct.ac.za University of Cape Town _______ Abstract This repository contains the code to pretrain the ELC-BERT architecture to the small WURA isiXhosa dataset, and subsequently finetunes the model on 3 finetuning tasks -NER, POS and NEWS, using the MasakhaNER organisation's isiXhosa datasets. ## Project Structure We illustrate the project structure below. Before we proceed, it is imporant to point out that in training ELC-BERT isiXhosa, we make use of code from numerous repositories. We use some files from these repositories with and without modeification. all repositories we use allow us the permissions to use them. References for all these externally used repositories are provided below. ## Setup Instructions ### 1. Install Required Packages Install the required Python dependencies by running: ```bash pip install -r requirements.txt ``` ### 2. Data download We provide the WURA xhosa datasets in the data/xhosa_raw folder. We upload this file directly as it contains a custom train/validation/test split. They dataset is also available at (castorini/wura) Before moving to the next step, first load the finetuning data for NER, POS and NEWS from MasakhaNER, MasakhaNEWS and MasakhaPOS resepctively by running the `python data_loader.py` file. This will download and save the data into the finetune_data folder. ### 2. Replicating our results Here, we provide a guide of the steps to train and finetune our models to replicate our results. Each section contains its own detailed README, and so the purpose of this section here is the provide a high level guide of the order to proceed. 1. preprocess the data. Navigate to the preprocess folder for instructions. 2. Create the tokenizer. Navigate to tokenizer folder for instructions. 3. cache our files for training. Navigate to the pre_training folder for instructions. 4. …