tylerachang/curse-of-multilinguality

When is multilinguality a curse? Language modeling for 250 high- and low-resource languages (EMNLP 2024). # curse-of-multilinguality Code for the paper, When is Multilinguality a Curse? Language Modeling for 250 High- and Low-Resource Languages (EMNLP 2024). Includes code for preparing pre-training (e.g. tokenizers and data) for multilingual language models. Language models are pre-trained using: github.com. We automate these scripts for our multilingual language modeling experiments. Pull the code using: git clone github.com git clone github.com We also use ```terashuf``` for shuffling large datasets: git clone github.com (cd terashuf && make) Python requirements are in ```requirements.txt```. Tested in Python 3.7. ## Dataset sources. Our text datasets are pulled from a variety of sources: | Dataset | Link | | ----- | ----- | | OSCAR | oscar-project.org | | Wikipedia | Wikipedia | | NLLB | huggingface.co | | NLLB Multi-Domain | nllb-multi-domain | | FLORES-200 | github.com | | Leipzig Corpora Collection | wortschatz.uni-leipzig.de | | eBible Translations | ebible.org | | AfriBERTa | castorini/afriberta-corpus | | Nusa | huggingface.co | | NusaX | huggingface.co | | Languages of Russia | web-corpora.net | | Evenki Life Newspaper | aclanthology.org | | Indigenous Languages Corpora | github.com | | AmericasNLP 2021 | github.com | | AmericasNLP 2022 | turing.iimas.unam.mx | | AmericasNLP 2023 | turing.iimas. …