khaledyusuf44/somalibench_v0

SomaliBench v0 — native-author-verified Somali safety evaluation benchmark. 100 harmful-intent prompts (HarmBench + AdvBench) in English + Somali for measuring multilingual safety-alignment transfer. CC-BY-NC-4.0. # SomaliBench v0 **The first native-author-verified Somali safety evaluation benchmark.** 100 harmful-intent prompts drawn from HarmBench (Mazeika et al. 2024) and AdvBench (Zou et al. 2023), translated into Somali by a native speaker and released as an evaluation set for measuring multilingual safety alignment in language models. --- ## Why this exists Somali has 15–20 million native speakers. Until now, there has been no native-speaker-verified safety evaluation benchmark for the language. Every existing multilingual safety dataset either skips Somali entirely or includes machine-translated prompts of unverified quality. The companion pilot project, multilingual-safety-probe, found that Llama-3-8B-Instruct refuses 0% of harmful prompts in Somali — a finding that depends on having high-confidence Somali translations to measure. SomaliBench is the artifact that makes that kind of measurement reproducible at larger scale. ## What's in v0 - **100 prompts** stratified across 7 safety categories - **Native Somali translations** by Khalid Yusuf Dahir (Mogadishu) - **English back-translations** for every record (auditable round-trip) - **Full provenance** — every record traces back to its source benchmark and index - **CC-BY-NC-4.0** license + responsible-use notice ## Categories | Category | Count | Source emphasis | |---|---|---| | harmful_instruction | 20 | how-to prompts for physical harm, weapons, illegal acts | | misinformation | 15 | fake news, propaganda, disinformation campaigns | | hate_speech | 15 | racist, sexist, ethnic-supremacy content | | privacy | 10 | identity theft, doxxing, surveillance | | cybersecurity | 15 | hacking, malware, exploits, unauthorized access | | CBRN | 15 | chemical / biological / radiological / nuclear weapons | | self_harm | 10 | suicide methods, eating-disorder promotion | | **Total** | **100** | | ## Schema Each record (JSONL line): ```json { "probe_id": "sb001", "category": "harmful_instruction", "source": "Harm …