charl-hendriks/MLSM-isiXhosa

# MLSM-isiXhosa This repository contains the source code and resources for training and evaluating a Masked Latent Semantic Modeling (MLSM) model specifically for the isiXhosa language. This work extends the research presented in the ACL Findings paper _Masked Latent Semantic Modeling: an Efficient Pre-training Alternative to Masked Language Modeling_ with the code available at the MLSM repository. It also utilizes the Masakhane Natural Language Understanding (NLU) evaluation sets for Named Entity Recognition (NER), Text Classification (NEWS) and Part of Speech tagging (POS) ## Creation of an isiXhosa model These are the steps followed to train and evaluate our isiXhosa MLSM model. Some minor changes were made to the "pretrainer.py" file from the MLSM repository, as well as the "train_ner.py", "train_textclass.py", "train_pos.py" from their respective Masakhane repositories. Therefore, we recommend any emulation of these results to utilize the versions of these files uploaded here. ### Step 1: Creating a custom tokenizer This step creates a custom tokenizer on the data that the model the model will be pre-trained on. ``` python train_tokenizer.py --vocab_size $VOCAB_SIZE --folder $INPUT_DATA_FOLDER --out_folder $TOKENIZER_NAME --cased ``` ```$VOCAB_SIZE``` = 25000 ```$INPUT_DATA_FOLDER``` = "wura-xh.tar.gz". ### Step 2: Creating an auxiliary model This step creates an isiXhosa language model trained on a traditional Masked Language Modelling (MLM) task with the bert-base architecture. This model provides semantic information for steps 3 and 4. ``` python pretrainer.py --transformer ${MODEL} \\ --reinit \\ --tokenizer ${TOKENIZER_NAME} \\ --out_dir ${AUXILIARY_MODEL_LOCATION} \\ --data_location ${PRETRAINING_DATA} \\ --training_seqs 25600000 \\ --batch 64 --grad_accum 16 ``` ```${MODEL}``` = "google-bert/bert-base-cased", the architecture to be used for this model. ```${TOKENIZER_NAME}``` refers to the tokenizer created in step 1 ```${PRETRAINING_DATA}``` …