bielng/Nuer-Tokenization-NLP

A SentencePiece tokenizer for Nuer (Thok Naath), a low-resource Nilotic language. Includes scripts to clean and prepare a parallel Nuer–English corpus, train a unigram subword tokenizer with full coverage of Nuer's diacritics (ŋ, ɛ, ɔ, ɣ, ɪ, combining marks), and verify lossless round-trip tokenization — built as a foundation for Nuer NLP/MT work. # Nuer Tokenizer A SentencePiece tokenizer pipeline for Nuer (Thok Naath), built from your `nuer_english_pairs.json` corpus (3,518 pairs). ## Setup ```bash pip install sentencepiece ``` ## 1. Prepare the corpus ```bash python prepare_corpus.py --input nuer_english_pairs.json --out-dir corpus --langs nuer english ``` This extracts the Nuer (and English) sentences, NFC-normalizes the text, expands "/"-separated spelling variants (e.g. `Jäli̱ kɛ mal/Ja̱lɛ kɛ mal.`) into separate training lines, dedupes, and writes `corpus/nuer.txt`. From your 3,518 pairs this yields **3,777 unique Nuer sentences** (more than the pair count, because of the expanded variants). ## 2. Train the tokenizer ```bash python train_tokenizer.py --input corpus/nuer.txt --model-prefix nuer_sp \ --vocab-size 2000 --model-type unigram ``` Produces `nuer_sp.model` and `nuer_sp.vocab`. Why these settings matter for Nuer specifically: - `character_coverage=1.0` — keeps every character, including `ŋ ɛ ɔ ɣ ɪ ä ë` and combining diacritics (e.g. the macron-below in `i̱`). A lower coverage would let SentencePiece treat these as "rare noise" and drop them — which for this orthography would be actively destructive. - `normalization_rule_name="identity"` — SentencePiece's default normalization is tuned for high-resource scripts and can interfere with combining marks. Since `prepare_corpus.py` already NFC-normalizes, we turn SentencePiece's own normalization off. - `vocab_size=2000` — sized for the ~3.8k-sentence corpus you currently have. Push it higher once you scale up the corpus (your 300k-pair corpus mentioned in earlier work would support 8k–16k+). **When you have your full ~300k-pair parallel corpus**, just point `--input` at that instead of this phrasebook-sized set for a more robust tokenizer. ## 3. Try it out ```bash python demo_tokenizer.py --model nuer_sp.model python demo_tokenizer.py --model nuer_sp.model --text "Ci̱ baak kɛ mal!" python demo_tokenizer.py --model nuer_sp.model --file …