Vitaly-Protasov/LR_Transfer

Experiments related to the transfer learning for extremely low-resource languages # Cross-lingual transfer between languages This repository is dedicated to experiments on cross-lingual transfer between high-resource and low-resource languages. We primarily focus on the case of extremely low-resource languages, examining how continued pretraining on high-resource languages affects model performance on low-resource languages in the Masked Language Modeling (MLM) task setup. Our goal is to investigate the feasibility of utilizing data from high-resource languages, training on it, and observing improvements compared to zero-shot performance when evaluating on unseen low-resource languages. Additionally, we evaluate the downstream performance of cross-lingual transfer in the context of the Machine Translation task. ## Install dependecies ```bash pip install jax==0.3.22 jaxlib==0.3.22 -f storage.googleapis.com pip install -r requirements.txt ``` ## Pipeline of experiments ### Dataset 1. Full raw dataset is available here: link 2. CLEAN dataset is available here: link 3. See resource_processing/ for information regarding resources and scripts that were used during dataset collection: * hzsk_processing * mc4_processing * ud_processing * wiki_processing * vk_hse_processing ### Cross-lingual transfer 1. Continued pretraining on high-resource languages: hr_training ```bash python hr_training.py --dataset_paths "English_files/" ``` 2. Evaluation of checkpoints on unseen low-resource languages: lr_evaluation ```bash python lr_evaluation.py --hr_lang "English" ``` 3. Vizualization of obtained results: notebook1, notebook2 ### Analysis using language and data features 1. Calculation of token intersection between languages: notebook 2. Usage of WALS features: notebook ### Downstream evaluation 1. Machine translation: link 2. POS tagging: notebook