mewaeltsegay/tokenizer

Tigrinya Language Tokenizers # Tigrinya Multi-Type Tokenizers for LLM Training A comprehensive collection of tokenizers specifically designed for the Tigrinya language, optimized for Large Language Model (LLM) training. This project provides three different tokenization approaches to suit various use cases. ## Overview This project provides three state-of-the-art tokenizers for Tigrinya text, each optimized for LLM training: 1. **BPE (Byte-Pair Encoding)** - Subword tokenization using merge operations 2. **WordLevel** - Complete word tokenization preserving linguistic boundaries 3. **SentencePiece** - Language-independent unigram subword tokenization All tokenizers are designed specifically for the Tigrinya language (Ge'ez script) and are fully compatible with popular machine learning frameworks. ## Tokenizer Comparison | Feature | BPE | WordLevel | SentencePiece | |---------|-----|-----------|---------------| | **Approach** | Subword merging | Complete words | Unigram probability | | **Vocab Size** | 32,000 | 50,000 | 32,000 | | **OOV Handling** | ✅ Excellent | ❌ Poor | ✅ Excellent | | **Word Integrity** | ⚠️ Partial | ✅ Perfect | ⚠️ Partial | | **Compression** | ✅ Good | ❌ Limited | ✅ Good | | **LLM Standard** | ✅ Common | ⚠️ Specialized | ✅ Very Common | | **Ge'ez Script** | ✅ Optimized | ✅ Optimized | ✅ Optimized | ### Which to Choose? - **BPE**: Best for general-purpose LLM training, good balance of compression and performance - **WordLevel**: Best for preserving Tigrinya word structure and linguistic analysis - **SentencePiece**: Best for production systems and multilingual models ## Features - **LLM-Optimized**: All tokenizers designed for modern LLM training - **Language-Specific**: Optimized for Tigrinya text and Ge'ez script - **Framework Ready**: Compatible with HuggingFace Transformers, PyTorch, TensorFlow - **Easy to Use**: Simple unified interface for training and usage - **Memory Efficient**: Handles large datasets with chunk-based processing - **Comprehensive Testing* …