Eddek1/Mabaan-nlp

Project date: 11/20/2025 via VS # Mabaan NLP Pipeline An NLP pipeline for documenting **Mabaan**, a low-resource Nilo-Saharan language spoken in South Sudan. The system uses retrieval-augmented generation (RAG) grounded in a Mabaan Bible corpus to assist linguists with glossing, lexicon building, and annotation — with a built-in human review workflow. ## The problem Mabaan has almost no digital linguistic resources. Documenting it requires linguists to manually gloss words, build a lexicon, and validate annotations — a slow and expensive process. This pipeline automates the bottleneck using an LLM, but keeps linguists in control through a human-in-the-loop review step. ## How it works ``` Raw field data (txt / csv / tsv / json) | v Ingestion layer Normalizes any supported format into structured Python dicts | v Bible RAG retrieval Fetches the most relevant Mabaan Bible verses as linguistic evidence | v LLM analysis (Claude) Glosses, translates, normalizes, or validates — grounded in attested text only | v Human review queue Linguists approve, reject, or correct each AI-generated entry | v Approved lexicon export ``` The key design principle: **Claude has no prior knowledge of Mabaan**. Every prompt injects retrieved Bible passages as the only allowed evidence source. If the model cannot ground a claim in an attested verse, it must say so and mark confidence as "low". ## Features - **Glossing** — breaks Mabaan words into morphemes using Leipzig Glossing Rules, with tone pattern detection (H/L/F/R) - **Lexicon extraction** — produces structured entries (headword, POS, English gloss, Arabic gloss, semantic domain, example sentence) from raw field notes - **Orthography normalization** — standardizes spelling variation against attested Bible forms - **IGT validation** — checks Interlinear Glossed Text for morpheme/gloss alignment and Leipzig convention compliance - **Translation** — produces English free translations with evidence verse citations - **Corpus summa …