ligeialugli/AsanteTwi_CorpusPreprocessingPipeline

Lemmatizer and PoS tagger for Asante Twi # Asante Twi Corpus Preprocessing Pipeline A reproducible pipeline used to preprocess *A Corpus of Asante Twi* (zenodo.org and asante-twi-corpus.fly.dev) and derive frequency list to be used for vocabulary testing in Ghana. ## Acknowledgements This work was made by Ligeia Lugli for the Luminos Fund thanks to funding from the Gates Foundation. Thanks to Najib Tackie for annotating all the datasets used to develop and evaluate this resource. --- ## Results Accuracy on the expert-annotated gold standard — **`gold/gold_final.csv`**, a Twi linguist's token-for-token lemma + coarse-PoS annotation of 13 documents (~10,400 words / 12,162 all-token), re-keyed to the delivered tokenization so it scores directly against the corpus with no alignment step. Run `python3 gold/eval_retok_gold.py`: | Split | N (words) | Lemma acc | Lemma bag-F1 | Coarse-PoS acc | PoS macro-F1 | |-------|-----------|-----------|--------------|----------------|--------------| | **Dev** (words-only) | 8,028 | 0.915 | 0.926 | 0.873 | 0.799 | | **Held-out** (words-only) | 2,344 | 0.902 | 0.908 | 0.827 | 0.788 | 5-fold spread (words-only; a confidence interval, not train/test CV — the pipeline never trains on the gold): dev lemma 0.915 ± 0.005 / PoS 0.873 ± 0.008; held-out lemma 0.902 ± 0.011 / PoS 0.827 ± 0.024. **Held-out documents are sealed** — never used to tune the pipeline — so held-out is the unbiased out-of-sample floor. Corpus: **~14M tokens / 5,650 documents** (86% doctrinal religious translation, set aside as domain-skewed; the vocabulary lists are built on the **~1.9M-token non-doctrinal** children + general subcorpora). Reproducibility: a fresh rebuild reproduces the deployed corpus **exactly — 0.0000%** gap, 0 / 14,014,166 tokens (`eval/repro_check.py`, runnable from any cwd). --- ## Pipeline Rule-driven first, then context-resolved. Each stage is idempotent and independently runnable. | # | Stage | Script | What it does | |---|-------|----- …