MehranLM-Tokenizer: A Natively-Trained Tokenizer for Sindhi Language Modelling

Sindhi, an Indo-Aryan language spoken by tens of millions of people, remains severely underrepresented in the tokenizers and training corpora of general-purpose multilingual Large Language Models. This underrepresentation results in poor fertility rates, high out-of-vocabulary rates (OOV rates), and degraded downstream performance whenever such models encounter Sindhi text. In this work, we introduce MehranLM-Tokenizer, a Byte-Pair Encoding tokenizer trained natively on Sindhi-language corpora, including Sindhi Wikipedia dumps, a Sindhi legal question-answering dataset, and a small self-typed corpus.  We describe our text-cleaning and normalization pipeline in detail and benchmark MehranLM-Tokenizer against eight other tokenizers -- four general-purpose multilingual tokenizers (mBERT, XLM-RoBERTa, Llama 3.1, Qwen2.5) and four natively-Sindhi tokenizers -- across three held-out datasets: a Sindhi sentiment-analysis corpus and a Sindhi misspelled-sentences corpus (in both misspelled and corrected form). We further present a variant, MehranLM-Tokenizer-With-Diacritics, which isolates the effect of diacritic normalization on fertility and byte-fallback behavior. We report fertility rate, compression rate, out-of-vocabulary rate, single-token retention rate, and byte-fallback rate for every tokenizer--dataset pair. MehranLM-Tokenizer achieves the lowest fertility rate and the highest single-token retention rate among all tokenizers tested, while incurring zero unknown-token (\texttt{}) events across all three datasets. Even though diacritics cause additional token splits, MehranLM-Tokenizer-With-Diacritics remains a close runner-up on nearly every metric.