zainmouhamed456-hub/whisper-pulaar-eval

Google Colab-ready Pulaar evaluation pipeline for fine-tuned Whisper models # Whisper Pulaar Eval A Colab-ready evaluation project for **fine-tuned Whisper models** on **Pulaar / Fulfulde** speech data using **Hugging Face Transformers**. This repository is a focused evaluation package for low-resource ASR work. It refactors the existing Codex-generated evaluation flow into a smaller, clearer structure that is easier to run in Google Colab and easier to share on GitHub. ## Pulaar Language Context Pulaar, also called Fulfulde or Fula in different regions, is a low-resource West African language. For ASR evaluation, small text inconsistencies can distort `WER` more than they would in higher-resource settings. In practice, Pulaar evaluation benefits from careful normalization that: - preserves meaningful characters such as `ɓ`, `ɗ`, `ƴ`, `ñ`, and `ŋ` - normalizes Unicode variants consistently - reduces punctuation-driven mismatch - keeps tokenization stable between predictions and references This project therefore reports both: - `raw_wer` and `raw_cer` - `normalized_wer` and `normalized_cer` ## Repository Structure ```text whisper-pulaar-eval/ |-- evaluation_notebook.ipynb |-- eval_script.py |-- README.md `-- requirements.txt ``` ## Colab-Focused Setup 1. Open `evaluation_notebook.ipynb` in Google Colab. 2. Switch the runtime to `T4 GPU`. 3. Run the notebook from top to bottom. The notebook installs the required Python packages, loads the evaluation script from this repo, fetches the Pulaar dataset from Hugging Face, and runs metric computation end to end. ## How To Load A Fine-Tuned Model `eval_script.py` accepts either: - a Hugging Face model id, such as `your-org/whisper-small-pulaar-ft` - a local checkpoint path, such as `/content/drive/MyDrive/models/whisper-pulaar-checkpoint` The notebook defaults to `openai/whisper-small` as a safe fallback so the pipeline is runnable immediately, but for real Pulaar evaluation you should replace it with your fine-tuned model id or checkpoint path. ## How To Run Evaluation ### In Cola …