samuelptah/kenyan-languages-corpus-engine

Automated preprocessing engine and parallel corpora generator for Kenyan regional languages (Kĩkũyũ, Kĩkamba, Luo, Kalenjin, Sheng, Kiswahili) . Find the repo on hagging face : huggingface.co # 🇰🇪 LughaGen Modular Engine LughaGen is a high-performance, modular data preprocessing and normalization engine designed to build high-quality parallel corpora for low-resource Kenyan and regional languages. The pipeline dynamically loads, normalizes, cleans, and partitions raw parallel datasets (CSV, Excel, Parquet, JSONL) into stratified train, validation, and test splits ready for Machine Learning, Neural Machine Translation (NMT), and LLM tokenizer training. --- ## 🚀 Key Features * **Dynamic Translation Unpacking:** Automatically detects and unpacks complex nested structures (like Hugging Face `translation` dictionary layouts). * **Self-Healing Schema Parsing:** If a dataset lacks a `metadata.json` file or has corrupted headers, the engine dynamically falls back to parsing folder-name patterns (e.g., `english-kamba` ➔ `eng` to `kam`) using a pre-configured language alias matrix. * **De-duplication & Sanitization:** Cleans formatting anomalies, stripping invisible artifacts (such as Windows Byte Order Marks and trailing web spaces). * **Stratified Splitting:** Handles balanced train/val/test dataset partitioning while maintaining strict index mapping integrity. --- ## 📊 Consolidated Corpus Statistics The pipeline successfully unified **6,190,120 parallel sentences** across several key regional language pairs: | Language Pair | Total Records | Unique Domains | Avg Source Length (Chars) | Description | | :--- | :--- | :--- | :--- | :--- | | **eng-kik** | 2,200,684 | 1 | ~47.9 | English ↔ Kikuyu (Gĩkũyũ) | | **eng-kam** | 1,679,956 | 1 | ~43.9 | English ↔ Kamba (Kĩkamba) | | **luo-eng** | 1,470,655 | 1 | ~52.0 | Dholuo ↔ English | | **kam-swa** | 223,350 | 1 | ~49.4 | Kĩkamba ↔ Swahili (Kiswahili) | | **kik-swa** | 219,495 | 1 | ~49.2 | Kikuyu ↔ Swahili | | **swa-eng** | 203,321 | 1 | ~48.9 | Swahili ↔ English | | **swa-kal** | 131,960 | 1 | ~50.5 | Swahili ↔ Kalenjin | | **eng-luo** | 40,422 | 1 | ~130.7 | English ↔ Dholuo | | **kam-kik** | 18,710 | 1 | ~ …