Telefonica-Scientific-Research/ASR-merging

Experiments on ASR models and LoRA merging strategies for low-resource languages adaptation and avoiding catastrophic forgetting # ASR-merging Research codebase for multilingual ASR fine-tuning and multiple-choice spoken question answering (MCQA), developed for the **MLC-SLM 2026 challenge** (Interspeech 2026) by the **Eloquence team** (Telefónica Scientific Research, Fondazione Bruno Kessler, CNR). The Eloquence team placed **5th** in the final ranking. The task (Task 2) requires selecting the correct answer (A–D) for questions grounded in multilingual spoken conversations spanning 21 languages. --- ## MLC-SLM 2026 Challenge: Submitted Systems Three independent systems were developed and submitted. The paper describing all three is in `tex/mlc-challenge/main.tex` (also published at Interspeech 2026). ### System 1 — Fine-tuned Voxtral-Mini-3B (LoRA + data augmentation) **Best Phase 1 accuracy: 0.7446** | Phase 2: 0.70 The fine-tuning pipeline has four components: **1a. Cross-lingual data augmentation** (`translate_*.py`, `deploy/run_translate_*.sh`) The MLC26 evaluation set contains ~52% cross-lingual question pairs (non-English audio, English question), while training data has 0% coverage. All non-English training questions are translated to English via **NLLB-200-distilled-1.3B**, masking audio quotations before translation. A *mixed* dataset concatenating original and translated copies of each non-English audio file best approximates the challenge distribution (original 0% cross-lingual → mixed 41.9% → target 52%): ```bash bash deploy/run_translate_challenge_nllb_1gpu.sh # translate challenge questions bash deploy/run_translate_training_nllb_1gpu.sh # translate training set bash deploy/run_translate_training_qwen3_1gpu.sh # alternative: Qwen3 translator ``` **1b. ASR transcript generation** (`transcribe_sessions.py`, `deploy/run_voxtral_transcribe_*.sh`) A dedicated Voxtral-Mini-3B LoRA ASR model is trained in two sequential stages: - **Seed model**: trained on MLC25 data (~29k utterances, 11 languages, 5 epochs, lr=5e-5). WER: 11.8% overall on MLC25 test set. Unseen ML …