ThilinaRajapakse/low-resource-language-models

# Low Resource Language Models ## Implemented Language Models 1. Kikuyu - [x] Baseline (Download) - [x] Baseline data + Bible Text (Download) 2. Ganda - [x] RoBERTa (Masked Language Modeling) (Download) - [x] GPT-2 (Language Generation) (Download) ## Setup 1. Install Anaconda or Miniconda Package Manager from here 2. Create a new virtual environment and install packages. `conda create -n transformers python pandas tqdm` `conda activate transformers` If using cuda:     `conda install pytorch cudatoolkit=10.1 -c pytorch` else:     `conda install pytorch cpuonly -c pytorch` 3. Install simpletransformers. `pip install simpletransformers` ## Usage ### Testing RoBERTa language models 1. Download the compressed model files from the link and extract to the `models/` directory. (E.g. `models/kikuyu_baseline`) 2. Run `test_language_model.py`. 1. Change the `language` variable to either `"kikuyu"` or `"ganda"` depending on the requirement. 2. Change the string in line 24 of `test_language_model.py` to test different sentences. The string may contain one ` ` token which the model will attempt to predict. ### Testing GPT-2 language generation 1. Download the compressed model files from the link and extract to the `models/` directory. (E.g. `models/ganda-gpt2`) 2. Run `test_language_generation.py`.