yigagilbert/hibiki-zero-finetuning_public

Training, evaluation, and diagnostic code for parameter-efficient fine-tuning of multimodal large language models for Luganda speech-to-speech translation — the research code behind the MSc dissertation of Gilbert Yiga (Department of Computer Science, Makerere University, 2026). # Luganda ↔ English Direct Speech-to-Speech Translation Training, evaluation, and diagnostic code for **parameter-efficient fine-tuning of multimodal large language models for Luganda speech-to-speech translation** — the research code behind the MSc dissertation of Gilbert Yiga (Department of Computer Science, Makerere University, 2026). The project adapted two architecturally distinct audio-language models to a synthetic parallel Luganda–English speech corpus: | Experimental line | Base model | Method | Outcome | |---|---|---|---| | **Hibiki-Zero** (this repo) | kyutai/hibiki-zero-3b-pytorch-bf16 | LoRA, then full fine-tuning (nine configurations) | No usable speech output; diagnosed a previously unreported, content-triggered **cb1–15 silence collapse** in the released base model on out-of-distribution Luganda Mimi codes | | **Step-Audio-2-mini** (separate repo) | stepfun-ai/Step-Audio-2-mini | LoRA (r=64, 154M params, 1.82%) | **BLEU 32.53 / COMET 0.717** on a matched 200-sample subset — approaching a Sunbird-tuned cascaded baseline (36.78 / 0.737) and exceeding it on WavLM SpeechBERTScore F1 | A bidirectional LoRA adapter additionally established direct **English → Luganda** translation (BLEU 11.16 vs 4.03 for the reverse-direction cascade). Live demo: ## Repository contents ### Training and configuration | File | Purpose | |---|---| | `train_sft.py` | Training entry point (single-GPU, bf16, gradient checkpointing) | | `sft_config.py` | Config dataclasses / loading | | `luganda_sft_v3.yaml` | **Stream-Layout-Corrected** configuration (full fine-tune, 40k max steps, peak LR 1e-5) | | `luganda_sft_v8.yaml` | **Contextually-Aligned** configuration (forced-alignment inner-monologue placement) | | `luganda_sft_v9.yaml` | **Rebalanced-Weights** configuration (cb weights [5, 2, …], text weight 1.4) | | `luganda_sft_v10.yaml` | **Cleaned-Corpus** configuration | | `luganda_sft_v4.yaml`, `luganda_sft_v5*.yaml`, others | Intermediate iterations documented in the di …