EbunoluwaArimoro/whisper-yoruba-asr-eval

# Whisper on Yorùbá — an ASR evaluation Evaluating OpenAI Whisper `large-v3` on Yorùbá speech using the FLEURS benchmark, to find out **where** general-purpose ASR breaks down on a tonal, low-resource African language — not just *whether* it does. The short answer: the model identifies Yorùbá perfectly and cannot transcribe it. Those two facts are separable, and the gap between them is the interesting part. --- ## Why this evaluation "Whisper is bad at African languages" is easy to assert and rarely measured. A single WER figure also hides the mechanism: a model can fail because it mishears sounds, because it loses tone, because it uses the wrong orthography, or because it hallucinates. Those have different fixes. This evaluation decomposes the failure into those parts, and adds a latency measurement — because accuracy is only half of what determines whether a model can actually serve users. ## Setup | | | |---|---| | **Model** | `openai/whisper-large-v3` | | **Data** | FLEURS `yo_ng`, test split (831 utterances), first 30 sampled | | **Hardware** | Single NVIDIA Tesla T4 (Google Colab) | | **Conditions** | Each clip transcribed twice — auto language detection, and with `language="yo"` forced | | **Reference** | `raw_transcription` (full Yorùbá orthography: tone marks and underdots preserved) | FLEURS reference transcripts were verified to carry complete diacritics before running, which is what makes the orthographic analysis below possible. ## Method Word error rate was computed four ways against progressively simplified orthography, to separate categories of error: 1. **Full orthography** — references and hypotheses as produced. 2. **Tone-insensitive** — combining grave, acute and macron (U+0300, U+0301, U+0304) stripped from both sides. Isolates tone errors while preserving the underdot, since `ọ`/`o` and `ẹ`/`e` are distinct vowels in Yorùbá, not tonal variants of one another. 3. **All diacritics stripped** — every combining mark removed from both si …