AI-for-Education/luganda-linguistic-benchmarks

# Luganda Linguistic Benchmarks These benchmarks were built as part of a project to fine-tune a small language model that assists teachers in Uganda with their daily tasks. Both are available in English and Luganda. - **LLPK — Luganda Linguistic Pedagogical Knowledge.** 100 multiple-choice questions on pedagogical content knowledge for early-grade literacy in Ugandan classrooms. - **LLK — Luganda Linguistic Knowledge.** 100 mixed-format questions (multiple-choice, true/false, short-form) testing the model's knowledge of Luganda itself. ## Datasets The datasets ship with this repo under `evals/` and are also published on Hugging Face: - LLPK — AI-for-Education/Luganda-Linguistic-Pedagogical-Knowledge-Benchmark - LLK — AI-for-Education/Luganda-Linguistic-Knowledge-Benchmark ## Quick Start ```bash git clone github.com cd luganda-linguistic-benchmarks cp .env.example .env # Edit .env with your API keys uv sync ``` ## Running the LLPK eval ```bash # Run a single model uv run luganda-linguistic-benchmarks run-llpk-eval --model claude-haiku-4-5-20251001 # Run all models in list_models.yaml uv run luganda-linguistic-benchmarks run-llpk-eval # Run on the Luganda dataset uv run luganda-linguistic-benchmarks run-llpk-eval --language luganda ``` Output: `results/summary_llpk_ .csv` (model, accuracy, bad_format). ## Running the LLK eval LLK is a two-step pipeline. Step 1 collects model responses; step 2 scores them (MCQ/true-false via regex; short-form via an LLM judge — Claude by default, so `ANTHROPIC_API_KEY` is required for step 2). ```bash # Step 1 — collect responses uv run luganda-linguistic-benchmarks run-llk-eval --model claude-haiku-4-5-20251001 # Step 2 — judge responses uv run luganda-linguistic-benchmarks judge-llk-eval --model claude-haiku-4-5-20251001 ``` Output: `results/summary_llk_ .csv` with overall, MC, short-form, and true/false accuracies. ## Adding a new model 1. Add an entry to …