tsuxalo/Spoken-Language-Translation-Model

A Framework for Translating Hausa Audio Recordings into English Text # Spoken-Language-Translation-Model Translating Hausa Speech to English — and Asking Whether Machine Learning Can Bypass the Need for a Written Language Collaborative Project with Nahom Azmach, Salim Gloyd, and Karun Mokha ## What this project does The goal is to take a recording of someone speaking Hausa and get back written **English** text of what they said. There are two fundamentally different ways to build this, and this project builds and compares **both**: - **A cascade** — two separate models chained together: one converts Hausa speech to Hausa text (Automatic Speech Recognition), the other translates that Hausa text to English. This is the project's main, fully-built pipeline — see **Part 1**. - **A direct model** — one model that goes straight from Hausa audio to English text, with no Hausa-text step in between. Explored as a smaller-scale pilot, specifically to test whether skipping the written-Hausa step avoids a weakness we found in the cascade — see **Part 2**. Both approaches start from the same idea: rather than building an ASR or translation model from nothing (which needs enormous amounts of data and compute), we take models already pretrained on speech and language in general — **Whisper** for speech, **NLLB** for translation — and adapt them specifically to Hausa. This is called **fine-tuning**. We're using the smallest version of Whisper (`whisper-small`) throughout, so training can realistically happen on a single consumer GPU rather than needing a data center. Below: general setup, then how each approach was built, then a head-to-head comparison of the two (**Part 3**). ## Where things stand right now - Environment, GPU/CUDA support, and all dependencies: set up and confirmed working. - **The cascade** (Hausa ASR fine-tune + NLLB-200 translation): fully built, trained, evaluated, and published. WER 44.7%, cascade translation BLEU ~8–10 on real ASR output. See **Part 1**. - **The direct approach**: a first trained pilot is done and pu …