lesego-mogable/isizulu-ensemble-asr

# isiZulu Ensemble ASR **A multi-agent ensemble Automatic Speech Recognition system for isiZulu — a critically under-resourced South African language.** > Research project completed in 2025 under the supervision of Prof. Duncan Coulter, University of Johannesburg. Part of a BSc Honours in Computer Science (Artificial Intelligence). --- ## Overview Automatic Speech Recognition (ASR) technology has largely failed to serve low-resource languages like isiZulu due to data scarcity and a historical focus on high-resource languages such as English and Mandarin. This project addresses that gap by designing, training, and evaluating a **multi-agent ensemble ASR system** for isiZulu. The system combines three architecturally distinct deep learning agents whose outputs are fused by a trainable GRU-based ensemble module, producing a single, more accurate transcription. --- ## Architecture Three independent ASR agents process input audio in parallel. Their logits are concatenated and passed through a GRU-based fusion module which learns to weigh and combine their predictions. ``` Input Audio │ ├──► Wav2Vec2 Agent (Fine-Tuned XLS-R) ──► Logits ─┐ ├──► Conformer Agent (Subset Fine-Tuned) ──► Logits ─┼──► GRU Fusion Module ──► CTC Decoder ──► Transcription └──► Custom CNN-RNN-CTC Agent (From Scratch) ──► Logits ─┘ ``` ### Agents **Wav2Vec2 (XLS-R)** — Fine-tuned from `facebook/wav2vec2-xls-r-300m`. Leverages self-supervised pre-training across 128+ languages, then adapted for isiZulu. Best performing individual agent. **Wav2Vec2 Conformer** — Initialized from `facebook/wav2vec2-conformer-large-960h` and fine-tuned on a subset of the isiZulu data. Combines convolutional layers for local feature extraction with self-attention for global context. **Custom CNN-RNN-CTC** — Built entirely from scratch on the NCHLT isiZulu corpus. Architecture: 2-layer CNN block → 3-layer bidirectional GRU (hidden size 512) → linear CTC classifier. Establishes a baseline for wh …