stefan027/pocket_polyglot

Pocket Polyglot Mzansi is a small 50M parameter machine translation model for South African languages. # Pocket Polyglot Mzansi Pocket Polyglot Mzansi is a small 50M parameter machine translation model for South African languages. The model is part of an ongoing research project that aims to develop a small ( 90% smaller than NLLB-200-600M, but sacrifices /devtest/ ``` By default, evaluations will be run for 30 translation directions (6 languages -> 15 languages pairs -> 30 total translation directions). The `-l` flag can be used to run evaluations for a subset of the six languages. For example, to run for English and isiXhosa only, run: ```bash python run_evaluation.py -m stefan7/pocket_polyglot_mzansi_50M_6langs -d /devtest/ -l "eng,xho" ``` The following outputs are generated: - `eval_results.csv`: The translations and labels for all evaluation examples. - `eval_results_summary.json`: The overall spBLEU, chrF, chrF++ scores. - `eval_results_by_language.csv`: The spBLEU, chrF, chrF++ scores for each translation direction. The `-o` flag can be used to write to different output files. ## Training To train the version of `Pocket Polyglot Mzansi` currently available on HuggingFace, make sure that the training data (`back_translations_6langs.txt`) is in the `./data` directory. The data can be downloaded from Kaggle. To start training, run: ```bash python train.py ``` To train with different configuration settings, create a new configuration file similar to `configs/default_config.json`. Then use the `-c` flag to train the model using custom settings: ```bash python train.py -c ``` ## Training the BPE tokenizer The pretrained tokenizer is provided in the repo. To train a tokenizer from scratch, see `scripts/train_tokenizer.py`.