NLP2CT/kNN-TL

[ACL 2023] kNN-TL: k-Nearest-Neighbor Transfer Learning for Low-Resource Neural Machine Translation # kNN-TL kNN-TL: k-Nearest-Neighbor Transfer Learning for Low-Resource Neural Machine Translation (ACL 2023) ## Overview Transfer learning is an effective method to enhance low-resource NMT through the parent-child framework. kNN-TL aims to leverage the parent's knowledge throughout the entire developing process of the child model. The approach includes a parent-child representation alignment method, which ensures consistency in the output representations between the two models, and a child-aware datastore construction method that improves inference efficiency by selectively distilling the parent datastore based on relevance to the child model. Training and Inference framework of kNN-TL. ## Installation ```bash cd kNN-TL pip install --editable . # python >=3.7 cd .. conda install faiss-gpu -c pytorch pip install sacremoses==0.0.53 ``` ## Data Preparation Download and preprocess the parent and child data ```bash # download and preprocess child data mkdir tr_en cd tr_en # donwload tr-en from drive.google.com # raw tr-en can be downloaded from opus.nlpl.eu cd .. fairseq-preprocess -s tr -t en --trainpref tr_en/pack_clean/train --validpref tr_en/pack_clean/valid --testpref tr_en/pack_clean/test --srcdict tr_en/dict.tr.txt --tgtdict dict.en.txt --workers 10 --destdir ${BIN_CHILD_DATA} # download and preprocess teacher data mkdir de_en cd de_en #donwload de-en from drive.google.com cd .. fairseq-preprocess -s de -t en --trainpref de_en/pack_clean/train --validpref de_en/pack_clean/valid --testpref de_en/pack_clean/test --joined-dictionary --destdir ${BIN_PARENT_DATA} --workers 10 ``` ## Training ### Parent Models ```bash cd train-scripts BIN_PARENT_DATA=${BIN_PARENT_DATA} # path of binarized parent data ## train for de-en bash train_parent.sh de en $BIN_PARENT_DATA ## train for …