translatorswb/African-ASR-benchmarking

Simple notebook for benchmarking voice datasets on various ASR models # African ASR Benchmarking This repository contains scripts for automated evaluation of ASR models on Next Voices Africa dataset. ## ✅ Evaluated Languages and Models (WER/CER %) | Language | MMS-1B-All | MMS-1B-FL102 | MMS-1B-L1107 | SeamlessM4T | Whisper | Xeus | |------------|------------|--------------|--------------|-------------|---------|------| | Zulu | 39.30/8.73 | 40.47/9.53 | ⚠️[^1] | ⏳ | ❌ | ⏳ | | Luo | | | | | ❌ | | | Kikuyu | | | | | ❌ | | | Yoruba | | | | | | | | Igbo | | | | | ❌ | | | Hausa | | | | | | | | Amharic | | | | | | | | Tigrinya | | | | | ❌ | | | Sidama | | | | | ❌ | | | Oromo | | | | | ❌ | | | Wolaytta | | | | | ❌ | | [^1]: `facebook/mms-1b-l1107` claims to support Zulu but raises a runtime error when 'zul' is specified. ## 📁 Files in this Repo | File | Description | |-------------------------------|--------------------------------------------------------------| | `ZA_African_Next_Voices_benchmarking.ipynb` | Notebook used for model inference and evaluation | | `asr_language_support_matrix.csv` | CSV matrix of ASR model support across selected African languages | | `hf_asr_scraper.py` | Script to scrape Hugging Face for ASR model support | ## 🔧 Dependencies Thi …