RayaneTaouaf/darija-codeswitch-eval

Native-reviewed Darija/French code-switching eval for LLMs. Gemini vs Claude comparison # Darija Code-Switch Eval **Status:** MVP complete — 60 items, native-reviewed, run against two frontier models (Gemini 3.1 Flash-Lite, Claude Sonnet 5), scored by an LLM judge with manual spot-checks. See Results below. ## Motivation Existing Darija benchmarks (DarijaMMLU, DarijaHellaSwag, DarijaBench, from the Atlas-Chat project; TerjamaBench from AtlasIA) evaluate models on clean, single-register input — a full question in Darija, or MSA, or Arabizi, but not mixed. Real Moroccan text messaging and speech constantly code-switches between French, Darija (Arabic script), and Darija (Arabizi/Latin script), often within a single sentence. A 2025 survey of 40+ Arabic LLM benchmarks flagged "cultural misalignment in translated datasets" and "insufficient multi-turn dialogue assessment" as open gaps — this project targets the code-switching piece of that gap directly. This is **not** a from-scratch benchmark. It extends existing work by testing a scenario (natural code-switching) that published Darija benchmarks don't cover. ## Method 1. 60 hand-written items spanning three categories (factual QA, reasoning, instruction-following) and four registers (French-Darija codeswitch, Darija Arabizi, Darija Arabic script, with control items in each pure register alongside the codeswitched ones). 2. Every item was drafted as an LLM first pass, then corrected by a native Darija speaker before being accepted into the set — the corrections themselves are logged in `data/style_notes.md` as a running grammar guide. 3. Ran the full set against two models from different families: Gemini 3.1 Flash-Lite and Claude Sonnet 5. 4. Scored every response with an LLM-judge pass (Claude, see caveat below), verdicting each as CORRECT / PARTIAL (right content, wrong register) / INCORRECT, cross-checked against a manual read of a subset. **Not done, flagged as future work:** a baseline rerun of an existing single-register benchmark (DarijaMMLU or TerjamaBench) against current frontier models, …