Psalms23Wave/AlkebulanAI

The first morphologically-aware tokenizer for Luganda, built from scratch.To give the African AI industry a real foundation to build on. # Alkebulan Bantu Language Tokenizer — Version 1 (ABLT v1) ### *The first morphologically-aware tokenizer for Luganda, built from scratch.* ### Powered by the **Sekan** algorithm — a Simpson-rule-based optimizer invented for agglutinative African languages. **Founder & Inventor:** Mathew Sekanjako **Owner:** Psalms23Wave LLC **Contact:** `psalms23wave.com` **GitHub:** github.com  •  **Hugging Face:** Psalms23Wave/Alkebulan-AI **Status:** v1.0.0 — validated, reproducible, open-sourced --- > ### Mission > **To give the African AI industry a real foundation to build on.** > Modern AI agents, translation systems, and language models for > African markets have been blocked at the tokenizer step because > every mainstream tokenizer is built for English. Alkebulan AI > changes that — starting with Luganda, extending by design to > every major African language family. If you are building AI for > Africa, this project is for you: fork it, extend it, ship it. > Just preserve the `NOTICE` file (Apache-2.0 §4(d)) so > the origin of the research stays traceable as the project grows. --- --- ## Why this project exists Modern tokenizers — BPE, WordPiece, standard SentencePiece — are trained on Western, predominantly English corpora. When I tried to fine-tune Mistral-7B on Luganda, they failed catastrophically. A single Luganda word like `Abanoonyiboobubudamu` ("refugees") was shattered into 9–11 meaningless sub-pieces. The model never saw the **prefix–root–suffix** structure that carries *all* the meaning in Bantu languages. Fixing the tokenizer alone was not enough. The gradients coming off those long agglutinated tokens spiked unpredictably, and every off-the-shelf optimizer (SGD, AdamW) amplified the spikes instead of damping them. So I built two things from scratch: 1. **ABLT** — a morphologically-aware tokenizer that *respects* Bantu structure before a single neural weight is touched. 2. **Sekan** — a Simpson-rule-based optimi …