OpenScit/polyglot-audiobench

Cross-lingual speech LLM evaluation: code-switching, accent robustness, low-resource ASR ## The Problem Most audio LLM benchmarks evaluate on clean English speech. The real world is messier: people switch languages mid-sentence, carry accents from their native language, and sometimes speak in low-resource languages that barely exist in training data. Current benchmarks don't catch when models fall apart in these conditions. **polyglot-audiobench** is an evaluation toolkit specifically designed to stress-test speech and audio LLMs on cross-lingual and multilingual scenarios. It does three things: 1. **Code-switching evaluation** — utterances that cross language boundaries, with boundary annotations 2. **Accent robustness probing** — paired audio/text across 20+ accents of the same content 3. **Low-resource language coverage** — 15 under-represented languages with standardised ASR and SQA tasks ## Our Approach The benchmark aggregates from existing datasets (CommonVoice, SEAME, FLEURS, MLS) but adds: - **Unified scoring**: a single cross-lingual capability score (CLCS) that aggregates across all three axes - **Stratified splits**: balanced by language family, accent region, and code-switch density - **Leaderboard infrastructure**: models submit via a CLI tool; results are verified and displayed on HF Spaces ## Show Me ```bash pip install polyglot-audiobench # evaluate a model polyglot-eval \ --model "openai/whisper-large-v3" \ --tasks code_switching accent_robustness \ --output_dir results/whisper-large/ # view scores polyglot-report results/whisper-large/ ``` Example output: ``` ┌─────────────────────────────────────────────────────────┐ │ polyglot-audiobench v0.3.0 — openai/whisper-large-v3 │ ├──────────────────────┬────────┬────────┬────────┬───────┤ │ Task │ Score │ Δ Base │ Rank │ Pass? │ ├──────────────────────┼────────┼────────┼────────┼───────┤ │ Code-switching (SEAME│ 42.1 │ +3.2 │ #4 │ ✓ │ │ Accent robustness │ 67.8 │ +8.1 │ #2 │ ✓ │ │ Low-resource ASR │ 31.4 │ -2.3 │ #7 │ ✗ …