oyinkanchekwas/low-resource-nlp-toolkit

Selective language routing and code-switch audits, evaluated on a reproducible AfriSenti benchmark. # Low-Resource NLP Toolkit Python tools for preparing and evaluating multilingual text, with an initial focus on African languages. The package covers Unicode-aware normalisation, lightweight language routing, code-switch audits, emotion-label mapping and classification metrics. Current release: `0.3.0`. ## What it covers - Text normalisation that preserves diacritics by default. - Lexical routing for Yoruba, Igbo, Hausa, Nigerian Pidgin, Swahili and English. - Selective routing, so weak decisions can be rejected and left unassigned. - Token-level code-switch audits with evidence spans and character offsets. - Emotion-label harmonisation across categorical and valence-arousal formats. - Classification, confusion-matrix and coverage-aware evaluation. - A command-line interface with no model download or API key requirement. ## Measured benchmark The router has been evaluated on the official AfriSenti development and test splits for Hausa, Igbo, Nigerian Pidgin, Swahili and Yoruba. The source revision and file hashes are pinned, and the source tweets remain outside this repository. On 18,402 held-out test tweets, ordinary corpus-language routing accuracy was 74.75%. A rejection threshold chosen on development data reached 89.65% accuracy on the accepted test items at 74.03% coverage. A paired stress test also measured the effect of removing diacritics. Overall accuracy fell from 74.75% to 69.87%; the largest changes were for Yoruba, down 15.17 percentage points, and Igbo, down 5.95 points. This is why the normaliser preserves diacritics unless removal is requested explicitly. See the full benchmark report for the method, per-language results, confidence intervals and limitations. ## Installation ```bash python3 -m venv .venv source .venv/bin/activate python -m pip install low-resource-nlp-toolkit low-resource-nlp --version ``` Route a text sample: ```bash low-resource-nlp route "abeg make una help me check this model output" ``` Reject routes with a narro …