KBrand26/Ufakazi

Ufakazi (isiZulu: "witness"). Auditing LLMs for language-based truthiness and legal-context bias in the South African setting Ufakazi Ufakazi (isiZulu: "witness"). Auditing LLMs for language-based truthiness and legal-context bias in the South African setting: given two contradicting, evidentially-balanced testimonies, does a model favor one based on the language it is written in? The languages under test are English, Afrikaans, isiXhosa, and isiZulu. Holding content and position constant, we render the same two testimonies into different languages and measure the model's forced credibility choice. Counterbalancing cancels content and position bias structurally, so what remains is the **language main effect**. For a full breakdown of our work: kbrand26.github.io See `DESIGN.md` for the experimental design and `CLAUDE.md` for architecture and conventions. Built on Inspect (`inspect_ai`, AISI's eval framework): each trial is an Inspect `Sample`, and a record-only scorer captures the model's forced choice rather than grading it (there is no correct answer). Real models go through OpenRouter by default (one API key for many providers); native `openai/` / `anthropic/` strings also work. ## Headline findings Interim panel of 8 models, 20 synthetic scenarios, full star design, 5 epochs (top-up to 10 was deemed too costly for the hackathon, so confidence intervals are wide-ish and these are not final): - **Only Claude Sonnet 4.6 and GPT-5.4 are bias-robust.** Their credibility preference is flat at chance across all four languages. - **Every other model tested shows significant pro-English bias**: Gemini 3.5 Flash, the full Gemma 3 ladder (4B / 12B / 27B), Grok 4.3, and (mildly) Qwen 3.7 Plus. The bias is **not** explained by capability (Gemini and Grok are frontier), by scale (it does not shrink across the Gemma ladder), or by a single lab (it spans Google, xAI, and Alibaba). - **A cross-linguistic gradient holds for every biased model**: Afrikaans bias is mild, isiXhosa and isiZulu severe. Human and machine Afrikaans agree closely, so machine translation is not …