charityking2358/multilingual-translation

Multilingual training techniques for performance improvement on low-resource languages # multilingual-translation This code evaluates neural machine translation benchmarks for low-resource languages using COMET and BLEU metrics. The bilingual framework uses parallel data from one high-resource language English and two low-resource languages Azerbaijani and Belarus. Our multilingual model boosts the performance of low-resource languages by using parallel datasets from similar languages that are higher resourced. This framework supplements our Azerbaijani dataset with Turkish and Belarusian with Russian. This work was derived from "When and Why are Pre-Trained Word Embeddings Useful for Neural Machine Translation?" ## Requirements * GPU environment that can run CUDA. We used an AWS Deep Learning AMI GPU PyTorch 1.10.0 (Amazon Linux 2) 20211115 with g4dn.xlarge instance * Approximately 100 GB volume or memory. This tutorial is helpful if you require more memory on your AWS volume after you've trained your models. ## Code ### Environment Setup Initialize your conda environment ``` conda create -n your_env python=3.8 conda activate your_env conda install pytorch torchvision torchaudio cudatoolkit=10.2 -c pytorch ``` This code requires the machine translation framework fairseq added to your environment ``` git clone git@github.com:pytorch/fairseq.git cd fairseq pip install . pip install --upgrade numpy export FAIRSEQ_DIR=`pwd` cd .. ``` Lastly, this code requires this repo to be cloned and requirements installed into your environment ``` git clone github.com cd multilingual-translation pip install -r requirements.txt ``` ### Data Download Our data consists of TED Talk data in 58 parallel languages to English. The raw data can be downloaded directly or by using the below command to run the download script. ``` python download_data.py ``` ### Bilingual Baselines These scripts train our neural machine translation systems only using parallel data from languages of interest. For our baseline, we will evalu …