amha-kindu/fidel-gpt

Training and fine-tuning pipeline for a custom GPT-style language model built exclusively for Amharic. Pretrained on a 12+ GB corpus and adapted on curated datasets, with support for SentencePiece tokenization, LoRA fine-tuning, and efficient inference tools. # Amharic Text Generation – Custom GPT Training This project provides the full training and fine-tuning pipeline for a **custom GPT-style language model** specialized for the **Amharic language**. Unlike most multilingual models where Amharic is a low-resource side case, this model is trained **from scratch on 12+ GB of Amharic text** and further adapted on **smaller supervised datasets** for task-specific improvements (dialogue, summarization, classification). ### 🔎 Why this project? - **Amharic is underrepresented** in large multilingual LLMs. Most perform poorly in fluency and grammar for Amharic text generation. - This repository builds a **dedicated model** with a vocabulary and training process tailored for the script, morphology, and punctuation rules of Amharic. - It enables downstream applications such as **chatbots, virtual assistants, educational tools, translation helpers, summarization systems, and Amharic NLP research**. ### 📌 What’s included - Scripts to **preprocess large Amharic corpora** (cleaning, tokenization, dataset sharding). - **Pretraining pipeline** (`train.py`) for building GPT models from scratch. - **Fine-tuning pipeline** (`finetune.py`) with support for LoRA adapters and parameter-efficient training. - **Tokenizer training** (`tokenizers/train_tokenizer.py`) to create a SentencePiece model with custom symbols and Amharic script coverage. - **Inference and chatbot scripts** (`inference.py`, `chatbot.py`) for interactive or batch text generation. - Utilities for **learning rate schedules**, **logging to TensorBoard**, and **LoRA parameter configuration**. ### 📊 Corpus & Training setup - **Corpus size**: 12+ GB raw Amharic text from multiple sources (books, Wikipedia, news, etc.). - **Vocabulary**: ~25,000 tokens trained with SentencePiece BPE. - **Architecture** (example): - Embedding dimension: 1024 - Number of blocks: 6 - Attention heads: 16 - Feed-forward dim: 4096 - Context window: 1024–2048 tokens - **Optimization**: AdamW with …