inesassia/FonBench

ASR benchmark and adaptation for the Fon language (low-resource African tonal language) # FonBench ASR benchmark and adaptation for the **Fon language** a low-resource African tonal language (~8M speakers in Benin). ## Overview This repository contains the code and results of a Master's thesis on automatic speech recognition for the Fon language. Six pre-trained multilingual ASR models are evaluated in zero-shot, and two adaptation strategies (LoRA and full fine-tuning) are compared on the best model. ## Results (test set, 3541 examples, 7 unseen speakers) ### Phase 1 : Zero-shot evaluation | Model | WER | CER | Status | |---|---|---|---| | **MMS-1b** | 83.44% | 31.88% | Retained for Phase 2 | | XLSR-53 | 101.03% | 64.42% | Evaluated | | OmniASR-CTC | 92.97% | 42.46% | Evaluated | | SeamlessM4T | 112.89% | 100.24% | Evaluated | | Whisper-small | 165.16% | 170.10% | Evaluated | | AfriHuBERT | 103.03% | 360.12% | Disqualified (CER>200%) | ### Phase 2 : Adaptation on MMS-1b | Configuration | Strategy | Params trained | WER | CER | |---|---|---|---|---| | Zero-shot | None | 0 | 83.44% | 31.88% | | LoRA-1 | LoRA r=8 | 2.0M (0.21%) | 32.20% | 9.08% | | LoRA-2 | LoRA r=32 | 7.9M (0.82%) | 28.75% | 8.18% | | FT-1 | Full FT (1e-5) | 967M (100%) | 24.32% | 7.17% | | **FT-2** | **Full FT (5e-5)** | **967M (100%)** | **19.27%** | **6.02%** | **Best result: WER 19.27% — a 64-point improvement over zero-shot baseline.** ## Resources - **Interactive leaderboard**: huggingface.co - **Best model (FT-2)**: huggingface.co - **LoRA model (r=32)**: huggingface.co - **Experiment tracking (W&B)**: wandb.ai - **Dataset**: Fongbe ASR ## Repository structure ``` FonBench/ ├── src/ # Source scripts │ ├── twer.py # T-WER composite metric │ ├── data_collator.py # Data collator for Wav2Vec2 │ ├── m …