ubayd-hattas/AfriGuard

A multilingual AI safety benchmark evaluating frontier LLM safety across South African languages through automated red-teaming. # AfriGuard: Multilingual Safety Red-Teaming for South African Languages > **Global South AI Safety Hackathon — Africa Track** > Cape Town Hub · 19–21 June 2026 > *Red-teaming frontier LLMs in six South African languages* --- ## 📋 Project Overview AfriGuard is the first comprehensive red-teaming evaluation framework targeting safety vulnerabilities in South African languages. We evaluate four frontier LLMs across **seven languages** (English, isiZulu, isiXhosa, Afrikaans, Sesotho, Sepedi, Tsonga) and **four harm categories** endemic to South Africa, measuring **Attack Success Rate (ASR)** — the percentage of adversarial prompts that bypass safety filters. **Key Finding:** Low-resource South African languages exhibit catastrophic safety failures. The mean ASR across all evaluations is **50.1%** — more than double the English baseline of **24.4%**. Certain model-language combinations reach **>90% ASR**, meaning models comply with harmful requests 9 out of 10 times. | Metric | Value | |--------|-------| | **Total Evaluations** | 1,120 (40 seeds × 7 languages × 4 models) | | **Languages** | 7 (6 African + English baseline) | | **Models** | 4 frontier LLMs | | **Harm Categories** | Financial fraud, xenophobic incitement, political disinformation, gang affiliation | | **Highest ASR** | 92.5% (Qwen 3 + Afrikaans/Sepedi) | | **Lowest ASR** | 7.5% (GPT-OSS + English) | --- ## 🎯 Problem Statement ### Why This Matters South Africa has **12 official languages** and a linguistically diverse population of 60+ million. LLMs are being deployed across mobile banking, government services (SASSA), and customer service — yet safety alignment is overwhelmingly concentrated on English. - **Language Inequality:** Communities speaking isiZulu (12M), isiXhosa (8M), Afrikaans (7M), Sepedi (5M), Sesotho (4M), and Tsonga (3M) face disproportionately higher exposure to AI-generated harm - **Real-World Impact:** Financial fraud via SMS/WhatsApp is endemic; gang recruitment, SASSA …