mohamad-755/african-tokenization-superbpe

Comparing SuperBPE against standard BPE tokenization across 9 African languages (Amharic, Hausa, Igbo, Chichewa, Kinyarwanda, Swahili, Wolof, Yoruba, Zulu) — measuring fragmentation reduction and downstream model impact. # Bridging the Tokenization Gap in African LLMs via SuperBPE Comparing **SuperBPE** (Liu et al., 2025) against a standard BPE baseline across 9 African languages — measuring whether "superword" tokenization meaningfully reduces text fragmentation, and whether that improvement translates into better model performance. **Program:** TRI AI Saturdays **Status:** Phases 1–4 complete. Phase 5 (evaluation) in progress. --- ## Why this matters Most tokenizers used in large language models are trained mostly on English text. When they encounter morphologically rich or agglutinative African languages, they fragment words into small, often meaningless pieces instead of the coherent morphemes a native speaker would recognize. This inflates token counts (raising compute cost) and degrades model quality (the model learns from broken pieces instead of stable units). This project builds a tokenization pipeline trained directly on native African-language text, and tests whether SuperBPE's approach — learning ordinary subwords first, then lifting the whitespace restriction to learn frequent multi-word "superwords" — reduces fragmentation compared to a matched standard BPE baseline. ## Languages Amharic (am) · Hausa (ha) · Igbo (ig) · Chichewa/Nyanja (ny) · Kinyarwanda (rw) · Swahili (sw) · Wolof (wo) · Yoruba (yo) · Zulu (zu) Spanning three language families (Niger-Congo/Bantu, Afro-Asiatic, Niger-Congo/Volta-Niger) to test whether results generalize rather than overfitting to one morphological pattern. See `docs/data_card.md` for corpus documentation, cleaning methodology, and known limitations (currently a stub — the full Data Card lives in Google Drive, see `docs/drive_structure.md`). ## Methodology ### Combined training corpus (temperature sampling, α = 0.3) Corpus sizes across the 9 languages vary by over 200x (Hausa: 46M words vs. Chichewa: 205K words). Naive proportional mixing would let large corpora dominate the learned vocabulary; naive equal weighting would ove …