jmanwillz/LLMs-and-African-Language

Fine-tuned XLM-RoBERTa on a Swahili dataset for masked language modeling and news classification tasks. Achieved a significant reduction in perplexity during pretraining, improving the model’s understanding of Swahili. Demonstrated higher accuracy and overall performance on Swahili news classification compared to the base model. # LLMs and African Language For this project you will be aiming to assist in performing systematic reviews. A systematic review is essentially a very structured literature review and our models will have two components: 1. a binary recommendation, 2. the extraction of relevant text. The binary recommendation is a label given to a document indicating whether the model believes it is a relevant piece of literature. A label of 1 indicates that it is relevant and 0 indicates that it is not. Secondly, pieces of relevant documents should then be highlighted if it contains important information. For your project, you must perform the systematic review in an African language other than English. To achieve this you must perform two full fine-tuning trainings of your model. The first is a more general fine-tuning of a pretrained model and can be anything you think will be helpful for the final performance of the model. The second fine-tuning is when you then train the model to identify your topic in the chosen language. You can also choose whichever topic you like - systematic reviews are generally done in medicine, however you will already be in a low-resource domain (which medicine is) by using an African language. So being low resource twice over is not a mandatory challenge. ## Specification Please find the specification here. ## Report Please find the report here. ## Running the Code Step 1: Create a conda environment with the required packages. ```bash conda env create -f environment.yml ``` Step 2: Activate the conda environment. ```bash conda activate development ``` Step 3: Login to Weights & Biases. ```bash wandb login --relogin ``` Step 4: Run the initial fine-tuning of FacebookAI/xlm-roberta-base on the MLM task. ```bash python fine_tune.py ``` Step 5: Run the second fine-tunings of the model on the classification task. ```bash # Trains on the xlm-roberta-base model. python classification.py --use-base ``` ```bash # Trains on the fine-tuned mode …