Tednn493/Cross-Lingual-Bias-in-Large-Language-Models-A-Comparative-Analysis-of-English-and-Swahili

# Cross Lingual Bias in Large Language Models: A Comparative Analysis of English and Swahili ## Description This project introduces a novel **Holistic Bias-like** dataset to systematically compare the generations of Large Language Models (LLMs) and examine their biases across languages. Specifically, this research investigates whether biases present in LLM outputs are amplified in Swahili compared to English, or whether these biases contextually change when prompts are presented in different languages. Inspired by the Holistic Bias dataset (Smith et al., "I'm sorry to hear that": Finding New Biases in Language Models with a Holistic Descriptor Dataset), this work extends bias evaluation across **9 demographic axes**: Ability, Nationality, Religion, Sexual Orientation, Ethnicity, Socioeconomic Status, Migration Status, Education Level, and Age. ## Research Questions 1. Do LLM biases manifest differently in Swahili versus English? 2. Are certain biases amplified or diminished when switching languages? 3. How does cultural and linguistic context influence bias in model generations across these demographic axes? 4. Do different sentence formats reveal varying levels of bias? 5. Do LLMs reason internally in English even when prompted in Swahili? ## Project Structure The project is organised into three independent experiments, each targeting a different angle of cross-lingual bias analysis. ### `sentence_completion/` — Cross-Lingual Sentence Completion & Bias Evaluation This is the core generation and evaluation pipeline. Sentence templates paired with demographic descriptors are substituted and sent to LLMs for completion in both English and Swahili. The completions are then evaluated for sentiment and semantic equivalence. **Models used:** - **Generation:** Gemini 2.5 Flash, ChatGPT 5.2 - **Evaluation (LLM-as-a-Judge):** Claude Sonnet 4.5 **What it measures:** - Whether LLMs produce more negative completions in one language over the other - Whether completi …