bnm-AI-dev/african-llm-safety-evaluation

Systematic red-teaming framework for multilingual LLM safety evaluation # African LLM Safety Evaluation Framework Achievement: Gold Medal Finish A systematic red-teaming framework for evaluating trust and safety failures in African large language models across 17 languages. Built for the Zindi African Trust & Safety LLM Challenge. **** --- ## Table of Contents - Motivation - Methodology - Key Findings - Technical Implementation - Results - Installation and Usage - Citation - Ethical Considerations - Contributing - License --- ## Motivation African AI deployment is accelerating. Models trained or fine-tuned on African languages are now being used in banking, healthcare, telecoms, and government services, serving populations in Swahili, Hausa, isiZulu, Amharic, and more. Yet the safety evaluation literature for these models is nearly absent. The gap is structural. Existing red-teaming benchmarks (AdvBench, HarmBench, SafetyBench) are English-centric and assume models with explicit safety fine-tuning. African-language models are typically developed under severe resource constraints, often without RLHF or safety fine-tuning. When safety properties are tested, they are usually evaluated in English, even for models whose primary deployment language is Swahili or Hausa. This framework addresses three specific gaps: 1. **No structured taxonomy for African-context risks.** Threats like mobile money fraud (M-Pesa phishing), ethno-religious misinformation, and traditional medicine misinformation are not covered by English-language safety benchmarks. 2. **No empirical attack surface mapping for completion-style African LLMs.** Models like Pawa-Gemma and N-ATLaS are primarily completion models, not instruction-tuned chatbots. Existing jailbreak techniques target instruction-following models and do not transfer. 3. **No cross-model consistency evidence.** Whether a safety failure in Swahili replicates in Hausa (same model, different language) or in Amharic (different model, same language family) is an open question. --- ## Methodolo …