rissalhedna/darija-llm-from-scratch

Train a Darija BPE tokenizer and a ~10M GPT from scratch. Karpathy-style, inline code, Wikipedia corpus (CC BY-SA 4.0). # darija-llm-from-scratch Build a Darija language model from scratch — tokenizer and GPT — in two hands-on notebooks you run and read end to end. ## Structure ``` darija-llm-from-scratch/ ├── darija_tokenizer.ipynb # Part 1 (~55 min, CPU) ├── darija_gpt.ipynb # Part 2 (~55 min, T4 GPU) ├── final/ │ ├── tokenizer.py # Clean BPE tokenizer (~200 lines) │ ├── model.py # Clean GPT transformer (~250 lines) │ └── train.py # Standalone training script (~100 lines) ├── data/ │ ├── darija.txt # Darija Wikipedia corpus (CC BY-SA 4.0) │ ├── extract_wiki.py # Reproducible Wikimedia dump extraction │ └── README.md # Corpus provenance and extraction ├── artifacts/ │ ├── darija_bpe_8k.json # Cached 8k merges │ ├── darija_tokens.bin # uint16 pre-encoded corpus cache │ ├── darija_tokens.meta.json # Corpus/token-cache integrity metadata │ ├── fertility_table.json │ ├── ablation_curves.json │ └── sample_outputs.json └── tests/ └── test_final.py ``` ## Quick start (Colab) The notebooks are opened from GitHub. The repo **must be public** — Colab's badge parses `github.com` and cannot see private files. After you push `main` to `github.com`: - Part 1 — Tokenizer - Part 2 — GPT (T4 GPU) The bootstrap cell clones the rest of the repo (corpus + cached artifacts) into `/content`. Re-running it is a no-op once those files exist. With `USE_CACHED = True` (default), both notebooks complete in under three minutes. Set `USE_CACHED = False` to retrain from scratch. ## Local setup ```bash pip install -r requirements.txt jupyter notebook darija_tokenizer.ipynb ``` ## What you will build **Part 1 — Tokenizer** - Why Arabic and Darija fragment badly under English-trained tokenizers - UTF-8 bytes, Unicode edge cases, Arabic normalisation - Byte Pair Encoding coded from scratch - An Arabic-aware regex split pa …