georgeedwardd/N-gram-Models

Character-level n-gram models and Byte-Pair Encoding for language identification across Afrikaans, Dutch, English, isiXhosa, and isiZulu using Wikipedia data. # Multilingual Language Modelling with N-Grams and Byte-Pair Encoding Statistical language modelling and language identification across five languages (**Afrikaans, Dutch, English, isiXhosa, and isiZulu**) using character-level n-gram models and Byte-Pair Encoding. ## Data All data comes from Wikipedia dumps and is unstructured. A shared preprocessing pipeline handles lowercasing, punctuation removal, accent stripping, and sentence boundary normalisation. ## Modelling Character-level trigram models are trained per language and used for language identification by scoring sentences via log-likelihood. A more robust interpolated n-gram model (unigram + bigram + trigram) is also implemented and evaluated on a held-out test set. The interpolated model also supports temperature-controlled text generation. ## Byte-Pair Encoding BPE merge rules are learned independently per language and compared pairwise to quantify how much subword structure is shared across languages. ## Structure ``` project/ ├── Ngrams.ipynb # Main notebook ├── utils.py # Preprocessing, n-gram models, BPE helpers ├── requirements.txt # Dependencies └── data/ ├── train.{af,nl,en,xh,zu}.txt ├── val.{af,nl,en,xh,zu}.txt └── test.lid.txt # Labelled test set for language identification ``` ## Setup ```bash pip install -r requirements.txt jupyter notebook Ngrams.ipynb ```