sara-graca/lowresource-xlsr-finetune

Fine-tuning XLS-R (wav2vec2-large-xlsr-53) for low-resource ASR, including dataset building, prediction and conversion to and from ELAN (.eaf) and Pangloss (.xml) formats. Built for Thulung but usable for other languages. # ASR Pipeline for Low-Resource Languages A toolkit for building automatic speech recognition (ASR) models for under-documented languages, built around fine-tuning XLS-R. It covers the full workflow: turning raw recordings + annotations into a clean training dataset, fine-tuning the model, and running inference on new audio with output formats ready for linguistic annotation tools. ## Pipeline overview raw audio + annotations -> make_dataset.py -> train/valid/test TSVs + clips train/valid/test TSVs -> fine_tune_xlsr_wav2vec2.py -> fine-tuned model new audio + model -> prediction_wav2vec2.py -> .tsv / .xml / .eaf / .txt transcripts ## Requirements - Python 3.8+ - Core: `torch`, `torchaudio`, `transformers`, `datasets`, `evaluate` - Data handling: `numpy`, `pandas` - Audio: `librosa`, `soundfile`, `pydub` - LM decoding (optional): `pyctcdecode`, `kenlm` ```bash pip install -r requirements.txt ``` --- ## 1. `make_dataset.py` — Build a training dataset Extracts sentence-level audio clips from recordings + their annotations, then splits them into train/valid/test sets at the **recording level** (all clips from one recording stay in the same split, so there is no data leakage). ### Step 1: Extract sentence clips ```bash python make_dataset.py create_audio \ --wav_dir /path/to/audio \ --trans_dir /path/to/annotations \ --out_dir /path/to/output \ [--language tdh] ``` Writes clips to `output/clips/` and a manifest `output/all.tsv`. Clips shorter than 1 s or longer than 20 s are dropped; audio is downmixed to mono and resampled to 16 kHz automatically. `--language` is optional (ISO 639-3 code). Without it, only a general text cleanup runs. With a code, a language-specific normalizer runs on top: - `tdh` (Thulung) - `nru` (Na) — also drops "fin peu audible" / "BEGAIEMENT" clips - `jya` (Japhug) — removes Chinese characters To add a language, define a `normalize_ ` function and register it in `NORMALIZERS`. ### Step 2: Spl …