bhoomi02-ai/Cross-Lingual-Hallucination-Drift-in-LLMs

We investigate whether cross-lingual hallucination drift is task-dependent, comparing factual QA (TruthfulQA) vs. commonsense reasoning (XCOPA) across English, Spanish, and Swahili using Aya Expanse 8B as the target model and GPT-4o-mini as judge. # Cross-Lingual Hallucination Drift in LLMs ### Does It Depend on Task Type? **Team:** Bhoomika Monthy Rajashekar, Devinn Chi, Chun Hsu, Anagha P Krishna **Course:** CS505 — Natural Language Processing, Boston University **Advisor:** Aaron Mueller --- ## Overview We investigate whether cross-lingual hallucination drift in LLMs is **task-dependent**. We evaluate **Aya Expanse 8B** across English, Spanish, Italian, and Swahili on two structurally different tasks — factual QA (TruthfulQA) and commonsense reasoning (XCOPA) — using GPT-4o-mini as an LLM-as-a-Judge evaluator. Italian appears in **both** benchmarks, enabling a clean within-language confound test: any difference in hallucination pattern between the two tasks, holding language constant, must be attributable to task type alone. --- ## Key Results ### Hallucination Rates | Task | Language | HR (%) | ΔHR vs EN | |------|----------|-------:|----------:| | TruthfulQA | English | 27.33 | — | | TruthfulQA | Spanish | 24.67 | −2.66 pp | | TruthfulQA | Italian | 28.67 | +1.34 pp | | XCOPA | English | 8.00 | — | | XCOPA | Italian | 9.33 | +1.33 pp | | XCOPA | Swahili | 98.67 | +90.67 pp | ### Drift Interaction Score (Φ) | Comparison | Φ (pp) | χ² | p | |---|---:|---:|---| | es/TruthfulQA vs sw/XCOPA | −93.33 | 45.44 | < 0.001 | | it/TruthfulQA vs it/XCOPA (within-language) | +0.01 | 16.98 | < 0.001 | **Interpretation:** XCOPA Swahili shows catastrophic drift (+90.67 pp) while TruthfulQA drift is negligible across all languages. The within-Italian test (same language, both tasks) confirms this is driven by task type, not language identity. ### Error Analysis | Task | Language | Incoherent | Wrong Answer | Total Hallucinated | |------|----------|-----------:|-------------:|-------------------:| | TruthfulQA | English | 0 | 14 | 41 | | TruthfulQA | Spanish | 1 | 11 | 37 | | TruthfulQA | Italian | 0 | 15 | 43 | | XCOPA | English | 0 | 5 | 12 | …