AymanELAKKAOUI/Darija-n-gram

# Darija Probabilistic N-gram Language Model This folder contains a small, self-contained training pipeline for a probabilistic Darija n-gram language model trained from the provided `data (1).rar` corpus. ## What it does - Reads the extracted corpus from `corpus/data` - Keeps all usable `.txt` content files and skips obvious tracker/noise files - Normalizes mixed-script Darija text - Evaluates multiple word-level n-gram orders on a held-out split - Selects the best order by validation perplexity - Retrains that model on the full dataset - Saves the trained model and a JSON training summary in `outputs/` ## Train ```powershell python train_ngram.py ``` Optional example: ```powershell python train_ngram.py --candidate-orders 2 3 4 --k 0.5 --min-freq 2 ``` ## Generate ```powershell python use_model.py --prompt "ana" --max-tokens 15 ``` ## Score text ```powershell python use_model.py --score "ana بغيت نمشي للسوق" ``` ## Outputs - `outputs/darija_ngram_model.pkl.gz`: trained model - `outputs/training_summary.json`: corpus stats, model choice, perplexity, and sample generations # Darija-n-gram