JehohshuaA/Yoruba-Diacritization-Benchmark

# Automatic Diacritization Models for Yorùbá — Evaluation Package This repository contains the evaluation code, model outputs, prompts, and scoring pipeline for the paper *"Automatic Diacritization Models for a High-Population Low-Resource African Language (Yorùbá)"* by J. I. Ayoola and P. O. Olukanmi (University of Johannesburg). ## Overview Four models are evaluated on the Yorùbá Automatic Diacritization (YAD) benchmark under the complete diacritic removal condition: | Model | Type | Access | |-------|------|--------| | Volta Baseline | Self-attention seq2seq (2018) | Open source | | mT5_base_yoruba_adr | Fine-tuned mT5-base | Open source (HuggingFace) | | GPT-5.2 | LLM | OpenAI API | | Gemini 3.1 Pro | LLM | Google AI Studio | Metrics: CER, WER, DER, WDER, BLEU, ChrF. ## Repository Structure - `data/` — undiacritized source and gold diacritized reference - `scripts/` — inference and evaluation code - `outputs/` — exact model outputs (3 runs each for the LLMs) - `prompts/` — verbatim prompts used for the LLMs ## Reproducing the Results ### 1. Open-source models The Volta Baseline and mT5_base_yoruba_adr outputs are in `outputs/`. mT5_base_yoruba_adr is publicly available on HuggingFace as `Davlan/mT5_base_yoruba_adr`. **Model snapshots used:** - GPT-5.2: `gpt-5.2-2025-12-11`, temperature 0 - Gemini 3.1 Pro: `gemini-3.1-pro-preview`, temperature 0 (via Google AI Studio) ## Notes on Reproducibility - All files are processed in UTF-8 with no Unicode normalisation. - The only post-processing is stripping leading/trailing whitespace. - LLM outputs are non-deterministic across runs even at temperature 0 due to internal reasoning; three runs are provided per LLM. ## Dataset The YAD benchmark is derived from the MENYO-20k test split. Original dataset: [github.com]. This study uses the complete diacritic removal condition (`yad_test.json`).