BehailuBerhanu/kambaata-llm-cultural-benchmark

A 77-item benchmark for evaluating cultural knowledge, hallucination, and epistemic behavior in large language models on Kambaata cultural knowledge. # Kambaata LLM Cultural Benchmark **When AI Doesn't Know Kembata: Hallucination, cultural substitution, and uncertainty in four general-purpose language models answering questions about an under-documented Ethiopian culture** Author: **Behailu Berhanu** Version: **1.0 candidate — consistency review pending** Benchmark: **77 manually verified questions × 4 models = 308 responses** Collection: **10–11 August 2026** ## What this repository contains This repository is the reproducibility package for the Kambaata LLM Cultural Hallucination Benchmark. The benchmark evaluates four general-purpose language models on fine-grained Kambaata cultural knowledge across 21 domains. Each response was scored on: 1. Accuracy (0–4) 2. Fabrication (None / Minor / Severe) 3. Narrative Substitution (0–3) 4. Geography Conflation (Y/N) The accompanying paper describes the methodology, findings, limitations, and implications. ## Repository structure ```text . ├── README.md ├── CITATION.cff ├── LICENSE-MIT ├── DATA-LICENSE-CC-BY-4.0.md ├── SCORING_RUBRIC.md ├── analysis.py ├── requirements.txt │ ├── data/ │ ├── benchmark_items.csv │ ├── ground_truth.csv │ ├── raw_responses.csv │ └── annotations.csv │ ├── paper/ │ └── when-ai-doesnt-know-kembata.pdf │ ├── results/ │ ├── recomputed_model_summary.csv │ ├── pairwise_wilcoxon.csv │ └── audit.json │ └── archive/ └── source_workbook_scored.xlsx ``` ## Reproduce the main analysis Install dependencies: ```bash pip install -r requirements.txt ``` Run: ```bash python analysis.py ``` The script checks the dataset structure before analysis. It expects exactly 77 items and 308 scored item/model rows, with 77 responses for each model. ## Data provenance The benchmark item bank was fixed before model querying. Ground truth was documented against the study's documentary source set and community knowledge. Raw model responses are preserved verbatim in `data/raw_responses.csv`; final annotations are in `data/annotations.csv`. …