chizkidd/igbo-asr-tonal-evaluation

Systematic evaluation of tonal fidelity in facebook/omniASR-CTC-1B when processing Igbo, a tonal Niger-Congo language with ~45 million speakers. # Igbo ASR Tonal Evaluation Systematic evaluation of tonal fidelity in facebook/omniASR-CTC-1B when processing Igbo, a tonal Niger-Congo language with ~45 million speakers. ## Overview This project reveals systematic tonal diacritic loss in a state-of-the-art multilingual ASR model: - **75.5% diacritic loss** on tonal markers (bootstrap 95% CI: [57.1%, 89.7%]) - **Minimal pair collapse**: Model cannot distinguish phonemically contrastive tones - **Orthographic bias**: Model hallucinates tone marks on monotone speech **Key Insight:** The model appears to generate diacritics probabilistically based on lexical priors rather than acoustic conditioning. ## Dataset **21 audio samples** across 4 error categories: 1. Cross-lingual Orthographic Interference (5 samples) 2. Phonemic Tone Sensitivity (6 samples) 3. Language Boundary Effects (5 samples) 4. Domain-Specific Lexical Coverage (5 samples) **View Dataset on HuggingFace** ### Listen to Examples Audio files are included in this repository (M4A format). Click to play directly on GitHub: **Tonal Minimal Pairs:** - 06_tonal_akwa.m4a - 4 different words collapsed to random outputs **Monotone Hallucination:** - 09_tonal_flat.m4a - Flat speech, model ADDED tones that weren't spoken **Code-Switching:** - 11_codeswitch_en2ig.m4a - English perfect, Igbo loses tones ## Quick Start ### Installation ```bash git clone github.com cd igbo-asr-tonal-evaluation pip install -r requirements.txt ``` ### Run Analysis ```bash jupyter notebook analysis.ipynb ``` Or open in Google Colab: ## Repository Structure ``` igbo-asr-tonal-evaluation/ ├── data/ │ ├── audio/ │ │ ├── 01_script_names.m4a # Cross-lingual interference samples │ │ ├── 02_script_formal.m4a │ │ ├── 03_script_numbers.m4a │ │ ├── 04_script_proverb.m4a │ │ ├── 05_script_slow.m4a │ │ ├── 06_tonal_akwa.m4a # Tonal minimal pairs │ │ ├── 07_tonal_oke.m4a │ │ …