AlokChedambath64/EfficientCPT-of-LLMs-for-LRLs

Implementation of the two core algorithms in the paper "Efficient Continual Pre-training of LLMs for Low-resource Languages. # EfficientCPT-of-LLMs-for-LRLs Implementation of the two core algorithms in the paper "Efficient Continual Pre-training of LLMs for Low-resource Languages. 1. Vocabulary.ipynb: Algorithm to choose the best tokens to extend your LLMs vocabulary with 2. Corpus.ipynb: Algorithm to choose the best sentence to pick from your corpus Adding tokens to your vocabulary/Adding sentences to your training data increase compute. These algorithms help you choose the most efficient additions of both. # Paper: arxiv.org # Corpus Selection Algorithm # Vocabulary Selection Algorithm