Kina-Research/English-Amharic-Code-Switching

EthioSwitch-Bench Overview EthioSwitch-Bench is the first large-scale, human-validated benchmark for evaluating Large Language Model (LLM) performance on English–Amharic code-switching in multi-turn conversational dialogue. Code-switching (CS) — the practice of alternating between two or more languages within a conversation — is a natural phenomenon among bilingual speakers, yet it remains a critical blind spot for most current LLMs. EthioSwitch-Bench addresses this gap by providing: - Richly annotated multi-turn dialogues with explicit language switch tags, switch-point tokens, coherence scores, fluency ratings, and inter-annotator agreement. - Two complementary benchmark tiers: a gold-standard hand-crafted set (v2.0) and a large-scale expanded corpus (v3.0). - Full reproducibility: every dialogue is deterministically generated or annotated, with pinned random seeds and a complete evaluation pipeline. Dataset Versions | Version | Dialogues | Topics | Turns | Source | |---------|-----------|--------|-------|--------| | v2.0 (gold) | 30 | 7 | 150 | Hand-crafted by expert annotators | | v3.0 (full benchmark) | 2,000 | 15 | 10,000 | 30 gold + 970 synthetic + 1,000 adapted from public corpora | 1. Clone the Repository git clone github.com /ethioswitch-bench.git cd ethioswitch-bench 2. Install Dependencies pip install -r code/requirements.txt 3. Load and Explore the Dataset Load the full benchmark (v3.0) with open("data/ethioswitch_bench_v3.json", "r", encoding="utf-8") as f: dataset = json.load(f) print(f"Dataset: {dataset['dataset_name']} v{dataset['version']}") print(f"Total dialogues: {dataset['total_dialogues']}") print(f"Splits: {dataset['split']}") Iterate over dialogues for dialogue in dataset["dialogues"][:3]: print(f"\nDialogue {dialogue['dialogue_id']} | Topic: {dialogue['topic']}") print(f" CS Frequency: {dialogue['code_switch_frequency']}") print(f" Dominance: {dialogue['language_dominance']}") for turn in dialogue["turns"]: print …