frhate/tiny-mamba-darija

A from-scratch implementation of a Mamba State Space Model (SSM) trained to speak Algerian Darija the colloquial Arabic dialect of Algeria, written in both Arabic script and Franco-Arab (Latin + numbers). Built with pure PyTorch. No Transformers. No attention. Just selective state spaces. # Tiny Mamba for Algerian Darija 🇩🇿 A from-scratch implementation of a **Mamba State Space Model (SSM)** trained to speak **Algerian Darija** — the colloquial Arabic dialect of Algeria, written in both Arabic script and Franco-Arab (Latin + numbers). Built with pure PyTorch. No Transformers. No attention. Just selective state spaces. > **"تعجبني هذيك اللقطة كي كنت ف المريكان . و الله غير يعطيك الصحة بالتوفيق"** --- ## 🚀 Features - **Pure PyTorch Mamba** — no `mamba-ssm` CUDA kernels required - **Custom BPE Tokenizer** trained on real Algerian social media text - **Mixed-script support** — handles Arabic script (`شحال`) and Franco-Arab (`3andek`) - **RTX 3060 Ti optimized** — trains on 8GB VRAM - **Repetition penalty + top-k sampling** for clean generation - **Resume training** from checkpoints automatically --- ## 📁 Project Structure . ├── darija_preprocessed.txt # Cleaned training corpus (not in repo, too big) ├── darija_tokenizer/ # Trained BPE tokenizer │ ├── tokenizer.json │ ├── vocab.txt │ └── special_tokens.txt ├── checkpoints/ # Saved model checkpoints (not in repo, too big) │ └── final_model.pt ├── darja_processing.py # Step 1: Clean raw text ├── train_tokenizer.py # Step 2: Train BPE tokenizer ├── mamba_darija.py # Step 3: Train Mamba model (pure PyTorch) ├── infernce.py # Step 4: Generate text ├── requirements.txt └── README.md --- ## 🛠️ Installation ```bash git clone github.com cd tiny-mamba-darija python -m venv venv source venv/bin/activate pip install -r requirements.txt ## 📚 Dataset Uses the Algerian-Darija dataset by Ayoub Kirouane. 23 MB of raw Algerian social media text Preprocessed to remove HTML, emojis, URLs, mentions, and duplicate characters 🏋️# Training from Scratch Step 1: Download & Preprocess python darja_processing.py Downloads from HuggingFace and cleans the text. Output: darija_preprocess …