Gacaca6/kinyarwanda-llm

An open Kinyarwanda (Bantu, agglutinative) GPT language model built from scratch — Kinyarwanda-trained BPE tokenizer (~1.94x more efficient than GPT-2) + corpus pipeline. Adapted from Raschka's LLMs-from-scratch. # Kinyarwanda LLM — a foundation for a model "for Rwandans" This project adapts ***Build a Large Language Model (From Scratch)*** by Sebastian Raschka into a starting point for a Kinyarwanda language model. The book gives a complete, language-agnostic GPT implementation (tokenizer → attention → transformer → pretraining → finetuning). This repo keeps that architecture and changes the two things that actually make a model work *for Rwandans*: **the tokenizer** and **the data**. ## The core insight A transformer doesn't care what language it models — the same math works for English or Kinyarwanda. What breaks for Kinyarwanda is everything *around* the transformer, above all **tokenization**. Kinyarwanda is a Bantu, **agglutinative** language: one word packs subject/object markers, tense, aspect, negation, and a stem. `ntibazabikora` ≈ "they will not do it." The book ships with GPT‑2's tokenizer, whose merges were learned on English, so it shatters Kinyarwanda words into meaningless fragments. This wastes context and makes learning harder. Peer‑reviewed work on the language — **KinyaBERT** (Nzeyimana & Niyongabo Rubungo, ACL 2022) — shows tokenization is *the* bottleneck for Kinyarwanda NLP. So the first, highest‑leverage adaptation is training the tokenizer on Kinyarwanda itself. ## What's been built and proven here A real Kinyarwanda BPE tokenizer was trained on authentic text (reconstructed from the MasakhaNER Kinyarwanda corpus) and measured against the book's default GPT‑2 tokenizer **on the same Kinyarwanda text**: | Tokenizer | tokens / word | chars / token | |---|---|---| | GPT‑2 (English BPE, book default) | 3.03 | 2.33 | | **Kinyarwanda BPE (ours)** | **1.56** | **4.53** | **≈1.94× more efficient — 49% fewer tokens per word.** And the splits are linguistically sensible: ``` abanyarwanda GPT-2 (5): ['ab','any','ar','w','anda'] ours (1): ['abanyarwanda'] ntibazabikora GPT-2 (6): ['nt','ib','az','ab','ik','ora'] ours (3): ['nti','baza','bikora'] i …