mohamad-755/magnet-african-tokenization

Reproducing and extending MAGNET (Ahia et al., NeurIPS 2024) to test multilingual tokenization fairness across African languages — evaluating whether script-level boundary predictors generalize to morphologically diverse Latin-script languages. # MAGNET: African Tokenization Fairness ## Project description This project reproduces and extends MAGNET (Ahia et al., NeurIPS 2024), a gradient-based tokenization method that uses language-script-specific boundary predictors to achieve equitable segmentation across languages. MAGNET's original evaluation covered 9 languages across Latin, Cyrillic, and Indic scripts — no African languages were tested. This project evaluates MAGNET on 9 African languages: Amharic (am), Hausa (ha), Igbo (ig), Chichewa/Nyanja (ny), Kinyarwanda (rw), Swahili (sw), Wolof (wo), Yoruba (yo), Zulu (zu) — spanning four language families (Niger-Congo/Bantu, Afro-Asiatic, Niger-Congo/Volta-Niger, Niger-Congo/Atlantic) to test whether results generalize rather than overfitting to one morphological pattern. Since most of these languages share Latin script (with Amharic as the exception, using Ge'ez script), this tests whether MAGNET's script-level boundary predictor design — built for structurally distinct scripts — actually achieves fairness across morphologically diverse languages that happen to share a script, or whether finer-grained adaptation is needed. See data_card.md for full corpus documentation, cleaning methodology, and known limitations. ## Background - Ahia et al. 2024, MAGNET (NeurIPS), openreview.net - Petrov et al. 2023, "Language Model Tokenizers Introduce Unfairness Between Languages" (NeurIPS), arxiv.org ## Project status Corpus cleaning complete for all 9 languages (see data_card.md). MAGNET is reproduced (`src/model/`) and trained (`src/training/train.py`), with three completed comparisons and one documented negative result: - **Script-level vs. per-language boundary-predictor routing** (both at a uniform target compression rate, β=0.5): giving each language its own predictor instead of sharing one per script did not improve cross-language fairness — coefficient of variation in bytes/segment was about 1.4x higher un …