farunawebservices/igala-gpt-from-scratch

Custom decoder-only transformer for Igala language generation built from first principles with multi-head attention and BPE tokenizer # ⚡ Igala GPT from Scratch A decoder-only transformer language model for Igala (low-resource Nigerian language) built entirely from first principles — no pretrained weights, custom architecture implementation. ## 🎯 Overview This project demonstrates **deep understanding of transformer architecture** by implementing a GPT-style model from scratch: - Custom multi-head self-attention mechanism - Positional encoding implementation - Layer normalization and residual connections - BPE tokenizer trained on Igala corpus - Autoregressive text generation **Why "from scratch"?** Most NLP projects fine-tune existing models. This project rebuilds the entire architecture to understand how transformers actually work under the hood. ## 🚀 Live Demo Try text generation: huggingface.co ## 🏗️ Architecture Model Configuration: Vocabulary Size: 5,000 tokens (custom BPE) Embedding Dimension: 256 Number of Layers: 6 Attention Heads: 8 Context Window: 128 tokens Total Parameters: ~12M ## 📊 Training Details - **Dataset**: 268KB Igala text corpus - **Training Steps**: 50,000 iterations - **Optimizer**: AdamW (lr=3e-4, weight decay=0.1) - **Hardware**: Single GPU (NVIDIA T4) - **Training Time**: ~8 hours - **Final Loss**: 2.34 (cross-entropy) ## 🛠️ Tech Stack - **Framework**: PyTorch (no HuggingFace Transformers) - **Tokenizer**: Custom Byte-Pair Encoding (BPE) - **Frontend**: Streamlit - **Deployment**: HuggingFace Spaces ## 📦 Installation ```bash # Clone the repository git clone github.com cd igala-gpt-from-scratch # Install dependencies pip install -r requirements.txt # Download trained model weights python download_model.py # Run the app streamlit run app.py 🔍 Usage Text Generation from igala_gpt import IgalaGPT, BPETokenizer # Load model and tokenizer model = IgalaGPT.load_pretrained("models/igala-gpt.pth") tokenizer = BPETokenizer.load("tokenizers/igala_bpe.json") …