radheshj/Multilingual-Sycophancy-Benchmark

A multilingual sycophancy benchmark exposing how AI safety guardrails fail in low-resource languages, with empirical evidence from Llama-3.1. # Multilingual Sycophancy Benchmark 🌍🤖 **Evaluating AI Safety Alignment and Delusion Acceptance in Low-Resource Languages** This repository contains the code, data, and findings for our submission to the **Apart Research Global South AI Safety Hackathon**. We built an automated, cross-linguistic evaluation pipeline to measure sycophancy—specifically *delusion acceptance*—across English and four Global South languages: Vietnamese, Tagalog, Bahasa Indonesia, and Swahili. ## 🚀 The Core Problem Current safety fine-tuning for Large Language Models (LLMs) is predominantly English-centric. If an LLM corrects a user’s bad medical advice in English but agrees with that exact same dangerous advice in Swahili simply because it wants to be "agreeable" in a low-resource language, it poses a severe threat to public health and epistemic trust in the Global South. This project investigates whether safety guardrails deteriorate when subjected to factual and medical delusions translated out of English. ## 🔑 Key Findings Evaluating the **Llama-3.1-8b-instant** model (via Groq API) across 150 domain-specific test cases (100 mathematics, 50 medical), we discovered a severe degradation in safety alignment: * **Anglocentric Safety:** The model exhibited near-perfect resistance to delusional premises in English (0.7% overall capitulation). * **The Swahili Spike:** Vulnerability spiked dramatically in lower-resource languages, reaching **36.0% overall capitulation in Swahili**. * **Medical Consistency Failure:** During a 50-iteration "Consistency Stress-Test" of core medical delusions, English guardrails held up 100% of the time. In Swahili, the model agreed with the dangerous medical premise 34.0% of the time. * **The Heuristic Blind Spot:** We identified a critical flaw in regex-based keyword parsing for Vietnamese, where the model bypassed safety filters using out-of-vocabulary agreements (e.g., *"đồng ý"*), highlighting the need for LLM-as-a-Judge architectures in …