project-acholi-mt24/project-acholi-mt24

This repository hosts code for a low-resource language machine translation project involving the Acholi language. The project is a part of the course Machine Translation (5LN711) at Uppsala University, Fall 2024. # project-acholi-mt24 This repository hosts code for a low-resource language machine translation project involving the Acholi language. The project is a part of the course Machine Translation (5LN711) at Uppsala University, Fall 2024. To clone the depository use: ``` git clone github.com ``` # Training the baseline model ### Step 1: Create virtual environment 1. In an appropriate directory (On UPPMAX), - Clone this repository - Run the script for creating a virtual environment: ``` bash create_mt_env.sh ``` 2. Activate the environment: ``` source ~/envs/activate_acholi_mt_env.sh ``` 3. To deactivate the environment: ``` deactivate ``` ### Step 2: Extract data for baseline model Run extract_data.py: ``` python extract_data.py ``` ### Step 3: Tokenize and clean the data Run preprocess.sh: ``` bash preprocess.sh ``` ### Step 4: Create vocabulary, Encode data using BPE, create .yaml-file with data configuration Run preprocess_onmt.py: Example: ``` python preprocess_onmt.py \ --train-src processed_data_moses/salt.train.tk.lc.clean.ach \ --train-tgt processed_data_moses/salt.train.tk.lc.clean.eng \ --dev-src processed_data_moses/salt.dev.tk.lc.ach \ --dev-tgt processed_data_moses/salt.dev.tk.lc.eng \ --src-lang ach \ --tgt-lang en \ --output-dir onmt_data \ --save-prefix data \ --src-vocab-size 7000 \ --tgt-vocab-size 7000 \ --src-min-frequency 2 \ --tgt-min-frequency 2 \ --src-bpe-operations 7000 \ --tgt-bpe-operations 7000 ``` ### Step 5: *Manually* create the file train_config.yaml Use the newly created file *data_config.yaml* as a base. Set parameters. See example-file: *train_config.yaml.example*. ### Step 6: Submit job to Snowy (Uppmax cluster): If needed update DATA_DIR in train_baseline_model.sh with the the full path to data directory ``` sbatch train_baseline_model.sh ``` ### Step 7 Preprocess test data Run script to preprocess the test set using the same encoding as for the test and dev set ``` python p …