jeffkingsley12/Tokeniser

High-performance, linguistically-aware Luganda BPE tokenizer. Features deterministic output, syllable-boundary preservation, and SIMD-optimized C implementation (SSE4.1/AVX-512/NEON) for ultra-fast processing. # 🇺🇬 Luganda Morphological Tokenizer A high-performance, production-grade tokenizer for the Luganda language, featuring a hybrid syllabification-based BPE approach, Re-Pair grammar compression, and a memory-optimized Split CSR LOUDS trie. ## 🚀 Key Features - **Linguistic Syllabification**: Luganda-aware syllable boundary detection with support for geminates, pre-nasalized consonants, and long vowels. - **Re-Pair Compression**: Iterative byte-pair encoding (BPE) variant optimized for morphology, delivering high compression ratios while preserving linguistic units. - **Split CSR LOUDS Trie**: A state-of-the-art, memory-mapped trie layout using Compressed Sparse Row (CSR) for O(1) fast-path lookups and minimal memory footprint. - **Aho-Corasick Truth Layer**: Integrated verification layer for identifying overlapping morphological matches and enforcing truth-set alignment. - **Zero-Copy Streaming**: Production-ready streaming API with lookahead margins and robust EOF handling. - **Memory Efficient**: mmap-based "page after page" corpus loading for multi-gigabyte training sets, supporting arbitrarily long lines (65KB+). ## 📊 Performance | Configuration | Cycles/Token | Tokens/Sec (3GHz) | Cycles/Byte | |---------------|-------------|-------------------|-------------| | **Regular** | 872 | 3.4M | 425.99 | | **Fused** | 1,267 | 2.4M | 431.02 | *Benchmarks on 9,389 documents, 670KB corpus. Fused path overhead includes real-time Aho-Corasick morphological annotation.* ## 🏗️ Architecture The tokenizer operates in a multi-stage pipeline: 1. **Syllabification**: Raw text is decomposed into a stream of Luganda syllables. 2. **Re-Pair Training**: Global frequency analysis identifies optimal merges to form subwords and tokens. 3. **Trie Construction**: Tokens are compiled into a LOUDS (Level-Order Unary Degree Sequence) trie using a Split CSR layout. 4. **Truth Layer Integration**: The Aho-Corasick automaton is b …