CARLOS-hub-droid/Chichewa-machine-translation

Master's thesis: Comparing LLMs and dedicated MT systems for English-Chichewa translation (AIMS-Cameroon 2026) # Machine Translation for Chichewa: Comparing LLMs and Dedicated MT Systems Master's Thesis (AIMS-Cameroon, 2025–2026) — Michaud Carlos Kouétsa Supervised by Dr. Dunstan Matekenya (The World Bank Group, Washington DC) ## 📌 Overview This thesis evaluates 13 machine translation systems for English→Chichewa translation (a Bantu language spoken by ~21.5 million people across Malawi, Zambia, and Mozambique), spanning four categories: commercial MT systems, commercial LLMs, open-source MT models, and open-source LLMs. The evaluation is conducted in two phases: - **Phase 1 — Baseline-status (zero-shot)**: all 13 models evaluated without adaptation. - **Phase 2 — Model adaptation**: in-context learning (3-shot / 5-shot) for the top open-source LLMs, and fine-tuning (full FT / LoRA) for the top open-source MT models. All models are evaluated on a domain-specific parallel corpus of 1,463 sentence pairs covering eleven domains (health, agriculture, politics, and more). ## 🏆 Key Results | Model | Setting | BLEU | chrF | |---|---|---|---| | Google Translate | Zero-shot | 27.04 | 57.20 | | NLLB-200 3.3B | LoRA Fine-tune | **20.91** | 56.10 | | GPT-5.5 | Zero-shot | 15.19 | 54.04 | | Helsinki-NLP | Full Fine-tune | 15.66 | 51.54 | | Gemma-3 27B | Zero-shot | 9.45 | 45.06 | **Main finding**: for Chichewa, domain-specific training data contributes more to translation quality than model size alone. A 74M-parameter model (Helsinki-NLP), once fine-tuned, surpasses a 3.3B-parameter model in its zero-shot state. ## 📂 Repository Contents - `02_opensource_MT.ipynb` — inference and evaluation of open-source MT models - `NLLB_FullFineTune_LoRA_Colab_...ipynb` — fine-tuning pipeline (full FT & LoRA) - `Benchmark_Visualization.ipynb` — results visualization and comparison plots ## ⚙️ Models Evaluated - **Dedicated MT (open-source)**: NLLB-200 3.3B, Helsinki-NLP Opus-MT, MADLAD-400 3B, mT5-base - **Open-source LLMs**: Gemma-3 27B, GPT-OSS 120B, Llama-4 Scout, Llama-3.3 70B, Mistral …