Uchebuzz/Naijaeval

Evaluation Metrics for African Languages # NaijaEval **Evaluation infrastructure for AI systems that mainstream benchmarks can't assess — built for African languages, code-switching, and dialectal robustness.** --- ## Why this exists Standard NLP benchmarks — GLUE, HELM, XTREME — were built for high-resource languages and standard dialects. When you build a system for Nigerian English, Yoruba, Igbo, Hausa, Nigerian Pidgin, or Swahili, none of those benchmarks tell you whether your system actually works. The specific gaps NaijaEval addresses: - **No metric exists for code-switch robustness.** A model that scores 0.85 on clean English may collapse when a user switches mid-sentence from English to Yoruba. - **No standard way to measure dialectal degradation.** WER on standard British English says nothing about WER on Nigerian English. - **Terminology preservation is unmeasured.** BLEU doesn't weight medical or legal terms differently from "the" — but in practice, getting "hypertension" wrong matters more than getting word order slightly wrong. - **Hallucination in low-resource translation is invisible.** When a model is undertrained on Swahili, it hallucinates. Standard metrics don't flag this. NaijaEval provides composable, task-agnostic metrics that work on real African language evaluation challenges — out of the box. --- ## Quickstart ```bash pip install naijaeval ``` ```python from naijaeval.metrics import ( CodeSwitchRateMetric, TerminologyPreservationMetric, HallucinationRateMetric, WERMetric, ) # Measure how mixed your test data is csr = CodeSwitchRateMetric() result = csr.compute( predictions=["I dey go market abeg, wetin be the price?"], references=[], ) print(f"Code-switch rate: {result.score:.3f}") # Code-switch rate: 0.444 # Check terminology preservation in medical translation tpr = TerminologyPreservationMetric(domain="medical") result = tpr.compute( predictions=["Alaisan naa ni malaria ati hypertension."], references=[], ) print(f"Term preservation: {result.score:.3f}") # Ter …