rifaasa/hassaniya-llm-benchmark

benchmark of modern LLMs on Hassaniya Arabic dialect » # hassaniya-llm-benchmark Code and evaluation data for a study of tokenization and zero-shot sentiment classification in Hassaniya Arabic (Mauritanian dialect, ISO 639-3 `mey`), compared against Modern Standard Arabic under a length-matched control. **The reusable part of this repository is the matching procedure.** `build_length_matched.py` takes a target corpus and a donor corpus and constructs a control set matched item by item on class and word count. It is not specific to Arabic: any dialect–standard comparison drawing on separately collected corpora faces the same confound. --- ## Why this exists Dialectal corpora are collected where dialect is written — social media, conversation. Standard-variety corpora come from edited sources — books, journalism, reviews. The two therefore differ in genre, register, length and label provenance all at once, and a difference measured across them is attributable to any of these. In our case the length asymmetry was a factor of 12.17 (5.67 words per text against 69.00). Controlling for it changed the conclusions: a mean fertility ratio of 1.093 fell to 1.008, and one of three models dropped out of the classification result. --- ## Contents | Path | What it is | |---|---| | `prepare_datasets.py` | Cleaning, deduplication, stratified splits | | `build_length_matched.py` | **The matching procedure**, with diagnostics | | `exp1_matched_check.py` | Fertility and subword token ratio, 8 tokenizers | | `exp2_sentiment.py` | Zero-shot classification via OpenRouter | | `exp2_metrics.py` | Per-class metrics, bootstrap intervals | | `exp2_recall_test.py` | Positive-recall test, Holm–Bonferroni corrected | | `exp2_followups.py` | Binary re-analysis, prediction distributions | | `exp3_forced_binary.py` | Forced binary choice on previously-neutral items | | `make_figures.py` | Manuscript figures | | `data/processed/hassaniya_processed.csv` | Hassaniya corpus, cleaned | | `data/processed/msa_labr_indices.txt` | Line indices into LAB …