Logo Lanfrica

graciax452/lute-shona

Domaine:

natural language processing

Type de record:

software
Créateur:
gra
Hôte:
# lute-shona A Shona language parser plugin for Lute3, splitting agglutinative noun/verb morphology into separately trackable tokens. ## What this solves Shona is space-delimited, so Lute's stock "Space Delimited" parser already gets word boundaries right. The problem is that a single space-delimited Shona word is often several grammatical morphemes glued together — e.g. `vachamubikira` = `va` (they) + `cha` (future) + `mu` (him/her) + `bik` (cook) + `ira` (for + ending vowel). Without splitting, the whole inflected word is one unclickable blob and you end up manually re-marking the root every time — the same pain point Korean particles caused before that parser was fixed. This plugin peels known noun-class prefixes and verb subject/tense/object/extension affixes off each already-space-split word, using hand-built lookup tables — no external tokenizer or dependency, since Shona doesn't need one for word boundaries. ## Design: lexicon-gated stripping A candidate affix is only stripped if what's left over matches a known root in a small seed lexicon (`rules.py`'s `NOUN_ROOT_LEXICON` / `VERB_ROOT_LEXICON`). This avoids **overstemming** — blindly stripping any string that looks like a prefix, even when it isn't grammatically that prefix, is a documented failure mode in rule-based agglutinative parsing (the same trap Swahili's SALAMA/xsma parsers are built to avoid). The practical effect: **most words outside the seed lexicons won't split in v1.** That's intentional, not a bug — a wrong split teaches wrong grammar, which is worse than showing the whole word. Grow the lexicons in `rules.py` as real reading text surfaces words worth adding. A handful of other small, tightly-gated tables handle known hard cases that don't fit the regular noun/verb pattern, all in `rules.py`: - `PROPER_NOUNS` — names (checked first; Shona names are often spelled identically to ordinary words, e.g. "Kuda" the name vs. "kuda" "to want") - `WORD_EXCEPTIONS` — whole-word bypasses for for …