mrtechzw/Shona-GPT

# ShonaGPT ShonaGPT is a lightweight, transformer-based language model trained specifically on the Shona language. It is designed to perform text generation and instruction-following tasks in Shona. This repository contains the model architecture, training scripts, and utilities for generating text with the trained model. > **Note:** The dataset is **not included** in this repository due to licensing/privacy reasons. You will need to provide your own dataset to train or fine-tune the model. --- ## Features - Causal language modeling tailored for Shona. - Lightweight GPT architecture with configurable depth and embedding dimensions. - Simple training and evaluation scripts. - Sample text generation utilities for testing the model interactively. - Implements tokenization using the GPT-2 tokenizer (`tiktoken`). --- ## Model Architecture ShonaGPT is based on a smaller GPT-like transformer: - Embedding size: 256 - Number of heads: 4 - Number of transformer blocks: 4 - Context length: 256 tokens - Dropout: 0.2 It includes standard components such as: - Multi-head self-attention - Feedforward layers with GELU activation - Layer normalization - Positional embeddings This design allows for efficient training on modest hardware while still capturing Shona language patterns. --- ## Installation 1. Clone the repository: ```bash git clone github.com cd ShonaGPT ```` 2. Install dependencies: ```bash pip install torch tiktoken ``` > Optional: Use a GPU for faster training and generation if available. --- ## Usage ### Text Generation ```bash python generate.py ``` * Enter a prompt when prompted (`input:`). * The model will generate Shona text continuing from your prompt. ### Training ```bash python train.py ``` * Modify `train.py` to point to your local Shona text dataset (`shona_small.txt` or similar). * Training parameters (batch size, learning rate, epochs) are configurable in `train.py`. * The model checkpoints are sav …