efesantos/lusophone-bias-audit

Four-arm behavioral bias audit of African (Angolan/Mozambican) Portuguese in frontier LLMs: EN / pt-PT / pt-BR / pt-AO, native-speaker-validated # Lusophone WEIRD-Bias Audit **To our knowledge, the first *behavioral* evaluation of African (Angolan/Mozambican) Portuguese in frontier LLMs** (an absence-of-evidence claim — see `docs/novelty.md` for the prior-art check and its date) — a four-arm paired design measuring whether models refuse more, answer less accurately, and answer less usefully on Africa-relevant content in Portuguese than in matched English, and whether **African Portuguese specifically** underperforms the European and Brazilian varieties. > **Honest framing.** We are *identifying and rigorously measuring a blind spot* — > African Portuguese falls between the low-resource-African-language safety > literature (TukaBench, XL-SafetyBench, LSR) and the well-served-Portuguese > literature (PoETa v2, P3B3, ALBA), touched only once, via perplexity, by Alves > (PROPOR 2024). We are **not** discovering that models fail. The pt-BR/pt-PT > contrast *replicates and extends* P3B3 (arXiv 2606.16753) and ALBA (arXiv > 2603.26516); the Brazilian-vs-European variety bias is established, not our > discovery. See `docs/novelty.md`. ## The design — a variant ladder Language is held fixed at Portuguese while the **variant** moves across the resource/region spectrum, so we can separate two pre-registered stories: | Arm | Variant | Role | |---|---|---| | EN | English | reference | | pt-BR | Brazilian Portuguese | best-resourced PT | | pt-PT | European Portuguese | Elmer's native variety | | pt-AO | Angolan Portuguese | the underserved cell — the headline | - **Brazil-centring:** both pt-PT and pt-AO underperform pt-BR (extends P3B3/ALBA). - **African-region penalty (the headline if found):** pt-AO underperforms *both* pt-PT and pt-BR — a residual penalty attached to the African variant/region itself, which resource level alone cannot explain. **Item set:** 4 types (F factual · V values · C cultural · R refusal) × 2 strata (Africa-relevant / Western-control) × 20 = **160 core items**, each rendered in 4 arms × …