Abel-Marie/amharic-gpt

A ~124M-parameter GPT-style LLM trained from scratch in PyTorch for Amharic text generation. # Amharic GPT — A Low-Resource LLM From Scratch > A GPT-style decoder-only Transformer trained **from scratch** in PyTorch for Amharic text generation. > ~124M parameters, XLM-RoBERTa tokenizer, end-to-end pipeline from raw text cleaning through training and sampling. Most foundation LLMs treat Amharic as a low-priority long-tail language. This repo: - reimplements every component of a GPT — attention, feed-forward, layer norm, transformer block, generator — without using `transformers.AutoModel`, - provides an Amharic-aware text-cleaning pipeline (Fidel character normalization, Geez numeral conversion, URL/handle stripping), - runs end-to-end with a single CLI: `preprocess` → `train` → `generate`. ## What this does - **Clean** raw Amharic text — Fidel character normalization, numeral conversion, URL/handle stripping, whitespace normalization. See `src/utils_text.py` and `src/data_preprocessing.py`. - **Tokenize** with the multilingual **XLM-RoBERTa** tokenizer (broad coverage of Geez script). - **Train** a **GPT-2-small architecture** decoder-only Transformer (12 layers, 12 heads, 768 emb dim, 256 context window) on the cleaned corpus, using **linear warmup + cosine LR decay**, **AdamW**, and **gradient clipping at 1.0**. - **Generate** Amharic text from a prompt with **top-k + temperature sampling** and **causal masking**. - **Log** training and validation loss curves and the learning-rate schedule. ## Architecture | Component | File | Notes | |---|---|---| | Multi-head causal attention | `src/attention.py` | Q/K/V projections, scaled dot-product, causal mask via `register_buffer` | | Feed-forward | `src/feedforward.py` | 4× expansion, GELU | | Transformer block | `src/transformer_block.py` | Pre-norm + residual; norm → attn/FF → dropout → residual | | Layer norm | `src/norm.py` | Custom impl with learnable scale & shift | | Decoder-only model | `src/gpt_model.py` | Token + positional embeddings, N stacked blocks, untied output head | | Loss | `src/loss.py` …