Rayanebjamai/n-gram_darija

# Darija N-gram Language Model This project trains a probabilistic word-level n-gram language model from the Darija corpus in `data 3`. ## What the code does - Reads the large corpus folder by streaming files line by line. - Skips metadata/noise files such as `.DS_Store`, `amlignore`, and `links_processed.txt`. - Normalizes URLs to ` ` and mentions to ` `. - Tokenizes Arabic script, Latin Arabizi text, numbers, hashtags, and punctuation. - Builds a vocabulary from the training split and maps rare words to ` `. - Trains a word trigram model by default. - Uses add-k smoothing so unseen n-grams still receive non-zero probability. - Evaluates on held-out lines and reports perplexity. - Saves the trained model and metrics under `models/`. ## Why trigram A trigram model, `P(w_i | w_{i-2}, w_{i-1})`, is a practical choice here: - unigram models ignore word order; - bigram models capture only one previous word; - trigrams capture short local Darija phrases while staying trainable on a laptop-size corpus. The default run uses `300,000` training lines and every 10th line as held-out test data. The folder has about `9.37M` text lines, so this is a deliberate sample that trains quickly while still using millions of tokens. ## Train ```bash python3 train_ngram.py ``` Main options: ```bash python3 train_ngram.py \ --data-dir "data 3" \ --n 3 \ --max-train-lines 300000 \ --max-test-lines 50000 \ --min-count 2 \ --smoothing 0.1 ``` To use more data, increase `--max-train-lines`. To train on the full corpus, pass a very large value, but expect a much larger model and longer runtime. ## Use the trained model Generate text: ```bash python3 use_ngram.py --generate ``` Score a sentence: ```bash python3 use_ngram.py --sentence "ana bghit nmchi l dar" ``` ## Current trained model The default training run produced: - model: `models/darija_trigram.pkl.gz` - metrics: `models/darija_trigram_metrics.json` - n: `3` - training lines: `300,000` - training tokens: `4,939,478` - …