gambigivesyouwiings/Mambo

Final year EEE project on S2S translation for the Swahili-English Language pair # Training a 100M Parameter Hibiki-Style Bidirectional En↔Sw Speech Translation Model on Kaggle ## Context The goal is to train a ~100M parameter end-to-end speech-to-speech translation model supporting **both English → Swahili and Swahili → English**, inspired by the Hibiki architecture (Kyutai, 2025). Hibiki uses a decoder-only multistream transformer that synchronously processes source and target speech via a Temporal Transformer + Depth Transformer, leveraging the Mimi neural audio codec for tokenization. The original Hibiki is 2.7B params (French→English); we must scale down to ~100M and train on Kaggle's 2× T4 GPUs (16GB VRAM each) with a **30hr/week GPU quota** (60hrs over 2 weeks) and 9hr max session length. **Key challenge:** English-Swahili is a low-resource pair — no large-scale parallel speech translation dataset exists. We must generate synthetic parallel data for both directions. **Bidirectional strategy:** Stages 0-2 (VITS training, text adaptation, audio pretraining) are **shared** — they are language-agnostic. Stages 3-4 (S2ST training + fine-tuning) are run **twice** — once per direction — branching from the same Stage 2 checkpoint. --- ## 1. Architecture Design (~100M Parameters) ### 1.1 Neural Audio Codec: Mimi (Frozen, pretrained) - Use the pretrained **`kyutai/mimi`** codec from HuggingFace (CC-BY license) - 12.5 Hz frame rate, **Q=8 codebooks** (reduced from 16 to cut depth transformer cost) - Codec is **frozen** — not counted in the 100M param budget - Encodes 24kHz mono audio → discrete tokens; codebook 1 = semantic, codebooks 2-8 = acoustic ### 1.2 Temporal Transformer (Main backbone, ~65M params) | Hyperparameter | Value | |---|---| | Latent dimension (d_model) | 512 | | FFN inner dim (gated SiLU) | 1408 | | Layers | 12 | | Attention heads | 8 | | Head dim | 64 | | Context window | 250 tokens (~20s at 12.5Hz) | | Local attention window | 250 tokens | - Per-layer params: self-attn (4 × 512²) + gated-FFN (3 × 512 × 1408) ≈ 3.2M - 12 …