Schadrackkarekezi/Small-KinyaGPT

KinyaGPT is a 51M-parameter decoder-only Transformer language model for Kinyarwanda, built from scratch with a custom tokenizer, full training pipeline, and text generation capabilities for a low-resource language. # Small KinyaGPT — A 51M-Parameter GPT Language Model for Kinyarwanda **KinyaGPT** is a **51 million parameter**, lightweight, decoder-only Transformer (GPT-style) language model trained from scratch on the mbazaNLP Kinyarwanda Monolingual Dataset. It demonstrates how to build a custom language model from the ground up including tokenizer creation, data preprocessing, model training, and text generation for a **low-resource language**. --- ## Highlights * ~51M parameters built entirely from scratch * Focused on Kinyarwanda, a low-resource Bantu language * Full end-to-end pipeline: tokenizer → preprocessing → training → generation * Metrics and training curves included * Interactive Jupyter notebook with the full training process --- ## Model Overview | Feature | Value | | ------------------- | -------------------------------------- | | **Architecture** | Decoder-only Transformer (GPT-style) | | **Parameters** | ~51 million | | **Embedding Size** | 512 | | **Layers** | 6 | | **Attention Heads** | 8 | | **Context Window** | 512 tokens | | **Vocabulary Size** | 32,000 | | **Dataset** | mbazaNLP Kinyarwanda Monolingual v01.1 | --- ## Training Performance | Metric | Value | | ------------------- | ----- | | **Training Loss** | ~3.95 | | **Validation Loss** | ~3.83 | | **Perplexity** | ~42.0 | | **Accuray** | ~33.44%| ### Training vs. Validation Loss --- ## How to Use ### 1. Install Dependencies ```bash pip install -r requirements.txt ``` ### 2. Train the Tokenizer ```bash python src/tokenizer.py --dataset mbazaNLP/kinyarwanda_monolingual_v01.1 --vocab_size 32000 --save_path data/kinyarwanda_bpe.json ``` ### 3. Preprocess the Dataset ```bash python src/utils. …