rwadataforum/kinya-performance

a repo of ideas to test the performance of AI models in Kinyarwanda # Kinyarwanda Performance Evaluations This repository include scripts to evaluate the performance of AI models in Kinyarwanda. It is still in early stage, but the long-term goal is to have a comprehensive evaluation framework. For the background please refer to VISION.md ## Latest Results ### 1. Kinyarwanda Reading Comprehension (KIN-RC Accuracy) KIN-RC tests whether **An AI model can read a paragraph and answer a question about it?** It is further explained in simple terms here docs/task_evals_KIN-RC.md Accuracy values come directly from `results/**/summary.csv`; See full leaderboard - Reading Comprehension for all models plus macro-F1. ### 2. Kinyarwanda Machine Translation (KIN-MT comet scorey) KIN-MT tests whether **an AI model can translate a sentence accurately between Kinyarwanda & English (in both directions)** It is further explained in simple terms here docs/task_evals_KIN-MT.md See the full leaderboard - Machine Translataion for all models with other evaluation metrics. ## Current Evaluations **(1) KIN-RC : KINYARWANDA READING COMPREHENSION** - `scripts/kin_rc_api.py` — runs KIN-RC on `facebook/belebele` (English, French, Kinyarwanda) and reports accuracy plus macro-F1 **(1) KIN-MT : KINYARWANDA MACHINE TRANSLATION** - `scripts/kin_mt_api.py` — runs KIN-MT on `openlanguagedata/flores_plus` across EN↔RW and EN↔FR sentence-level directions with chrF, BLEU, and optional SSA-COMET Both scripts share helpers in `kin_eval/` (API clients, metrics, IO) and write results to `results/ / / /` as CSV + JSONL summaries. ## Repository Layout ```text . ├── README.md # High-level overview (this file) ├── VISION.md # Benchmark strategy and tiers ├── RUNBOOK.md # How to install and run evaluations ├── docs/ │ └── task_evals_KIN-RC.md # Plain-language task descriptions ├── kin_eval/ # Shared helpers (clients, metrics, IO) ├── scripts/ │ ├── kin_mt_api.py # Translation evaluation entr …