KulmichaBullo/rendille-translator

Speech-to-speech translation for Rendille language (East Cushitic) using Whisper ASR + NLLB MT + Coqui TTS # Rendille → English Machine Translation Model A production-ready neural machine translation system for Rendille (rel), an East Cushitic language of Kenya (~60K speakers). ## 📋 Project Status **Current Phase:** Phase 1 — Data Collection & Baseline - ✅ Project plan documented in `PROJECT_PLAN.md` - ✅ Data collection scripts created - ⏳ Awaiting: Rendille Bible corpus download validation - ⏳ Next: Baseline NLLB evaluation & tokenizer expansion ## 🏗️ Architecture ``` Rendille English NMT Pipeline ├── Data Layer │ ├── eBible Corpus (BibleNLP) — Verse-aligned parallel Bible │ ├── JW300 Dataset (OPUS) — Potential parallel source │ └── Synthetic data — Back-translation via Somali/Oromo pivots ├── Model Layer │ ├── Base: facebook/nllb-200-distilled-1.3B (if Rendille in vocab) │ └── Extended tokenizer (if Rendille NOT in NLLB) → expand embeddings ├── Training Layer │ ├── LoRA fine-tuning (rank r=16) — parameter-efficient │ ├── Mixed precision training (fp16/bf16) │ └── Cross-lingual multi-task (Somali+Oromo+Rendille) optional └── Evaluation Layer ├── BLEU, chrF++ (on held-out Bible books) ├── COMET (neural metric) └── Human evaluation (fluency/adequacy) ``` ## 🚀 Quick Start ### Prerequisites - Python 3.11+ - GPU (NVIDIA CUDA) recommended for fine-tuning; CPU okay for inference - Internet access for downloads (~5–10 GB) ### Installation ```bash # Clone / enter project cd ~/rendille-translation # Install dependencies (prefer editable) pip install -e . # Or manually: pip install torch transformers datasets peft accelerate sacrebleu sentencepiece requests ``` ### Step-by-Step Workflow #### 1. Check NLLB Language Coverage ```bash python scripts/check_nllb_langs.py ``` - If Rendille is listed → mark `--add-token` = False in training - If NOT listed → we MUST expand tokenizer + embedding layer #### 2. Download Rendille Bible Corpus ```bash python scripts/download_bible.py ``` Expected output: ``` Found Rendille (rel) — ~27,000 verses Saved rel_vre …