sbourziq1337/Darija-GPT

# Darija GPT A tiny GPT-style language model built from scratch in C++17, trained on Moroccan Darija (Moroccan Arabic) text. No PyTorch, no TensorFlow — just raw C++ with a custom autograd engine. --- ## What is this? This project implements a complete transformer-based language model (GPT architecture) entirely in C++ from the ground up. It includes: - **Custom autograd engine** with forward/backward passes - **Byte-level tokenizer** that handles any language natively - **Multi-head self-attention** with causal masking - **Transformer blocks** with LayerNorm and residual connections - **GELU activation** and feed-forward networks - **Cross-entropy loss** with numerically stable softmax - **SGD optimizer** with gradient clipping - **Text generation** with temperature, top-k, and repetition penalty - **Model save/load** in a custom binary format - **Interactive chat mode** ### Architecture ``` Input Tokens | v +---------------+ | Embedding | Token embeddings + Position embeddings +---------------+ | v +---------------+ +---------------+ | Transformer | --> | Transformer | --> ... (x N layers) | Block 1 | | Block 2 | +---------------+ +---------------+ | v +---------------+ | LayerNorm | +---------------+ | v +---------------+ | Linear (LM | Language modeling head | Head) | +---------------+ | v Logits -> Softmax -> Next Token Prediction ``` --- ## Quick Start ### Prerequisites - C++17 compatible compiler (g++ or clang++) - `make` - A text file with Darija (or any) text at `data/darija.txt` ### Build ```bash make ``` This creates two executables: - `./llm` — Training and generation - `./chat` — Interactive chat mode ### Train ```bash ./llm ``` The model will: 1. Load `data/darija.txt` 2. Train for 10,000 steps (default, configurable in `src/main.cpp`) 3. Save weights to `models/darija_gpt.bin` 4. Generate sample text from prompts ### Chat ```bash ./chat models/darija_gpt.bin ``` Commands: - `/quit …