aadilganigaie/LLM-for-Low-Resource-Languages

Extending the Vocabulary of Large Language Models for Low-Resource Languages In the realm of natural language processing (NLP), large language models (LLMs) have revolutionized various applications, from translation to text generation. However, most LLMs are English-centric, with limited support for low-resource languages like Kashmiri. This limitation arises primarily due to the scarcity of available corpora for these languages, making it challenging to train robust models. One effective approach to address this issue is to extend the vocabulary of existing LLMs. By leveraging the emergent behavior of LLMs, we can enhance their performance on downstream tasks like translation for underrepresented languages. This article demonstrates how to extend the vocabulary of an LLM for Kashmiri and how this method can be adopted for other low-resource languages. The Problem with Pretrained Tokenizers Using pretrained tokenizers for underrepresented languages often leads to more token usage, which increases computational costs and power requirements. For instance, using a pretrained tokenizer for Kashmiri results in a higher number of tokens, leading to inefficiencies. To overcome this, extending the vocabulary of the existing LLM is beneficial. Extending the Vocabulary Step 1: Train a New Tokenizer We start by training a new tokenizer specifically for Kashmiri using the SentencePiece library. Step 2: Load and Test the New Tokenizer After training the new tokenizer, we load it and test its performance on Kashmiri text: Length of kashmiri text: 33 -------------- NEW TOKENIZER -------------- Length of encoded IDs: 6 --- Compression ratio: 0.18 --- Encoded token IDs: [385, 464, 1800, 12474, 1125, 826] --- Decoded text: اگر أس باقی اؠجِکؠشن سِسٹم وُچھو -------------- MISTRAL TOKENIZER -------------- Length of encoded IDs: 37 --- Compression ratio: 1.12 --- Encoded token IDs: [1, 28705, 28915, 29461, 28947, 28705, 28915, 31865, 29008, 28705, 28983, 28915, 29115, 28975 …