skogsgren/MT-5LN711-Runyankole

Low-Resource Neural Machine Translation For Runyankole; project for MT-5LN711 @ Uppsala University # Runyankole Low-Resource Machine Translation Project for MT-5LN711 at Uppsala University. ## Prerequisites In order to run model training, we first have to get our dependencies and our data. To get our dependencies simply use `pip install` (preferably in a `venv` or `conda` environment): ```{bash} pip3 install -r requirements.txt ``` The data is available unprocessed on `futurum` at `/home/gugre01/data.tar.xz`. Copy that file to the `./src` directory. To get it processed a preprocessing pipeline is available, written as a shell script. The preprocessing pipeline makes sure to process the data in the correct order (see details below). To run the preprocessing pipeline run: ```{bash} cd src ./preprocessing_pipeline.sh ``` You will end up with a folder inside `src` called `data` with the following structure: ``` . ├── all │   ├── eng.bpe │   ├── eng_dev.bpe │   ├── eng_dev.detok │   ├── eng_test.bpe │   ├── eng_test.detok │   ├── eng_train.bpe │   ├── eng_train.detok │   ├── nyn.bpe │   ├── nyn_dev.bpe │   ├── nyn_dev.detok │   ├── nyn_test.bpe │   ├── nyn_test.detok │   ├── nyn_train.bpe │   └── nyn_train.detok ├── backtranslated │   ├── eng.bpe │   ├── eng_dev.bpe │   ├── eng_dev.detok │   ├── eng_test.bpe │   ├── eng_test.detok │   ├── eng_train.bpe │   ├── eng_train.detok │   ├── nyn.bpe │   ├── nyn_dev.bpe │   ├── nyn_dev.detok │   ├── nyn_test.bpe │   ├── nyn_test.detok │   ├── nyn_train.bpe │   └── nyn_train.detok ├── bible │   ├── eng.bpe │   ├── eng_dev.bpe │   ├── eng_dev.detok │   ├── eng_test.bpe │   ├── eng_test.detok │   ├── eng_train.bpe │   ├── eng_train.detok │   ├── nyn.bpe │   ├── nyn_dev.bpe │   ├── nyn_dev.detok │   ├── nyn_test.bpe │   ├── nyn_test.detok │   ├── nyn_train.bpe │   └── nyn_train.detok └── original ├── eng.bpe ├── eng_dev.bpe ├── eng_dev.detok ├── eng_test.bpe ├── eng_test.detok ├── eng_train.bpe ├── eng_train.detok ├── nyn.bpe ├── nyn_dev.bpe ├── nyn_dev.detok ├── nyn_test.bpe ├── nyn_test.detok ├── nyn_train.bpe └── nyn_ …