HafsatiMohammed/darija-asr-training-evaluation

# Darija ASR Training & Evaluation (Whisper) This project builds and evaluates Moroccan Darija ASR models under compute constraints using Whisper. It includes: - dataset preparation + filtering + text normalization, - local fine-tuning for Whisper models, - detailed evaluation with global and bucketed metrics. --- ## 1) Project Structure - `data_preparation_with_text_normalization.py` Downloads/merges source datasets, normalizes Arabic text, applies usability filtering, and exports a Whisper-ready dataset. - `train_whisper.py` Fine-tunes a Whisper checkpoint (commonly `openai/whisper-small` or `openai/whisper-base`) on the prepared dataset. - `evaluation_whisper.py` Runs evaluation and exports: - global metrics (WER/CER/exact match), - per-sample details (JSONL + CSV), - grouped bucket summaries, - listening sets. - `requirements.txt` Minimal dependencies for training/evaluation. --- ## 2) Environment Setup ```bash cd /home/mohammed/Documents/darija-asr-training-evaluation python -m venv .venv source .venv/bin/activate pip install -U pip pip install -r requirements.txt ``` My exact conda env is given ### Hugging Face token (required) > **Important:** Export your HF token before data prep / model access. ```bash export HF_TOKEN="your_huggingface_token_here" ``` (Optional, but useful when pulling private assets/checkpoints): ```bash export HUGGINGFACE_HUB_TOKEN="$HF_TOKEN" ``` --- ## 3) Data Preparation ```bash python data_preparation_with_text_normalization.py ``` By default, outputs are written under: - `./darija_merged_analysis/` - Whisper-ready dataset path used by training/eval: `./darija_merged_analysis/whisper_ready_dataset_16k_paths_success_only` --- ## 4) Train a Local Model Edit constants in `train_whisper.py` as needed: - `MODEL_NAME` (e.g., `openai/whisper-small` or `openai/whisper-base`) - `OUTPUT_DIR` (e.g., `./whisper-small-darija`) - batch size, max steps, etc. Then run: ```bash python train_whisper.py ``` Saved artifacts inc …