Logo Lanfrica
  • Home
  • Atlas
  • Insights
  • Docs
  • Sign in

© 2026 Lanfrica. All rights reserved. All copyrights of the resources shown on the Lanfrica website belong to the original copyright holders, unless explicitly stated otherwise.

Surfacing Subtle Stereotypes: A Multilingual, Debate-Oriented Evaluation of Modern LLMs

Domain:

natural language processing

Record type:

paperdataset
Creator:
SaeAbdShe
Host:avatar
Large language models (LLMs) are widely deployed for open-ended communication, yet most bias evaluations still rely on English, classification-style tasks. We introduce \corpusname, a new multilingual, debate-style benchmark designed to reveal how narrative bias appears in realistic generative settings. Our dataset includes 8{,}400 structured debate prompts spanning four sensitive domains -- Women's Rights, Backwardness, Terrorism, and Religion -- across seven languages ranging from high-resource (English, Chinese) to low-resource (Swahili, Nigerian Pidgin). Using four flagship models (GPT-4o, Claude~3.5~Haiku, DeepSeek-Chat, and LLaMA-3-70B), we generate over 100{,}000 debate responses and automatically classify which demographic groups are assigned stereotyped versus modern roles. Results show that all models reproduce entrenched stereotypes despite safety alignment: Arabs are overwhelmingly linked to Terrorism and Religion ($\geq$89\%), Africans to socioeconomic ``backwardness'' (up to 77\%), and Western groups are consistently framed as modern or progressive. Biases grow sharply in lower-resource languages, revealing that alignment trained primarily in English does not generalize globally. Our findings highlight a persistent divide in multilingual fairness: current alignment methods reduce explicit toxicity but fail to prevent biased outputs in open-ended contexts. We release our \corpusname benchmark and analysis framework to support the next generation of multilingual bias evaluation and safer, culturally inclusive model alignment.

Visit

arxiv.org

Languages

Swahili

Tags

Computation and LanguageComputers and Society

Similar

Cross-Lingual Auto Evaluation for Assessing Multilingual LLMsmSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text TasksSurfacing a hidden literaturemBLIP: Efficient Bootstrapping of Multilingual Vision-LLMsLLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual FeedbackControlling Language Confusion in Multilingual LLMs

Cross-Lingual Auto Evaluation for Assessing Multilingual LLMs

Evaluating machine-generated text remains a significant challenge in NLP, especially for non-English

mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks

Large Language models (LLMs) have demonstrated impressive performance on a wide range of tasks, incl

Surfacing a hidden literature

Scholars throughout the world are working to diversify the knowledge base in educational leadership

mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs

Modular vision-language models (Vision-LLMs) align pretrained image encoders with (frozen) large lan

LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback

To democratize large language models (LLMs) to most natural languages, it is imperative to make thes

Controlling Language Confusion in Multilingual LLMs

Large language models often suffer from language confusion, a phenomenon in which responses are part