michsethowusu/llm-afro-leaderboard

Testing the accuracy of LLMs when translating to and from African languages. # LLM Afro Leaderboard LLM Afro Leaderboard is an open tool for evaluating how well Large Language Models (LLMs) handle translations to and from African languages—where data is often scarce and existing benchmarks fall short. Our aim is simple: build a transparent, reproducible way to test zero-shot translation accuracy, so that researchers and developers can push forward inclusive, high-quality language technologies. ------ ## Table of Contents - About the Project - Current Results & Limitations - Evaluated Languages and Models - How to Contribute - Technical Requirements - Evaluation Methodology - License - Acknowledgements ------ ## About the Project African languages remain among the least represented in today’s AI landscape. Training data is limited, and benchmarks are even rarer. LLM Afro Leaderboard sets out to change that by providing a practical, standardized way to measure how well different LLMs can translate African languages. By building shared benchmarks, we can highlight both progress and gaps—ultimately contributing to fairer, more accessible language technology across the continent. ------ ## Current Results & Limitations The initial results in this repo were generated on a small, exploratory dataset: - 15 paragraphs per language (from news, literature, and academic texts) - 47 languages tested - 14 different models evaluated Limitations to note: - Results are anecdotal, not definitive benchmarks. - Many proprietary models weren’t tested. - Scaling across multiple languages/models is resource-intensive. Still, this early test provides a valuable starting point for deeper work. ------ ## Evaluated Languages and Models The following 47 African languages were evaluated: Afrikaans (afr), Akan (aka), Amharic (amh), Bambara (bam), Bemba (bem), Dinka (dik), Dyula (dyu), Ewe (ewe), Fon (fon), Fulah (ful), Hausa (hau), Igbo (ibo), Kabiyè (kbp), Kamba (kam), Kikuyu (kik), Kimbundu (kmb), Kinyarwanda (kin), Kongo (kon), Lingala (lin), Ga …