abssisophia9-hub/darija-ngram-lm

# Darija N-gram Language Model 🇲🇦 A character-level n-gram language model trained on Moroccan Darija (Arabic dialect) text data. Built from scratch using pure Python with no external ML libraries. --- ## Project Structure ``` darija-ngram-lm/ ├── prepare_data.py # Cleans and combines raw text sources into corpus ├── train.py # Trains the n-gram model (memory-safe streaming) ├── evaluate.py # Evaluates model quality using perplexity ├── requirements.txt # No external dependencies needed ├── src/ │ ├── preprocessing.py # Text cleaning utilities │ ├── ngram_model.py # N-gram model core logic │ └── evaluation.py # Evaluation helpers └── data/ # (not included — see Data Sources below) ``` ## How It Works This project builds a trigram character-level language model over Moroccan Darija text: ### 1. prepare_data.py — Data Preparation - Walks through all raw .txt source files in named subfolders - Cleans text: keeps only Arabic script characters (U+0600–U+06FF) - Removes duplicates using MD5 hashing - Writes a unified data/corpus.txt ### 2. train.py — Model Training - Reads the corpus in 4KB chunks (memory-safe streaming) - Counts character trigrams (sequences of 3 characters) - Periodically prunes low-frequency n-grams to prevent memory overflow - Saves the trained model as models/model.json ### 3. evaluate.py — Model Evaluation - Loads the trained model - Computes perplexity on a Darija test string - Uses Laplace smoothing for unseen character sequences --- ## Setup No installation needed. Just make sure you have Python 3.8+: python --version All libraries used are part of Python standard library: json, math, os, re, collections, hashlib. --- ## How to Run Step 1 — Prepare the corpus python darija-ngram-lm/prepare_data.py Step 2 — Train the model python darija-ngram-lm/train.py Step 3 — Evaluate the model python darija-ngram-lm/evaluate.py --- ## Results | Metric …