Tanguy1902/malagasy-stemmer

Ultra-fast Malagasy NLP stemmer & tokenizer in Rust & Python. Powered by Finite State Transducers and two-level morphology # malagasy-stemmer High-performance morphological stemmer, tokenizer, and search filter for the Malagasy language ( Teny Malagasy ). [ Documentation en Français ] --- ## Overview **`malagasy-stemmer`** is a native Rust library with Python bindings designed to bridge a critical gap in Natural Language Processing (**NLP**) and Information Retrieval (**IR**) for the Malagasy language (*Teny Malagasy*, an Austronesian language spoken by 30+ million people). Malagasy features a rich, agglutinative morphology with complex nasal mutations, infixation, reduplication, and consonant sandhi in compound words. `malagasy-stemmer` accurately reduces inflected surface forms to their canonical roots (*fototeny*). --- ## Key Features - **Accurate Morphological Normalization (*Fototeny*)**: - **Nasal mutations**: `manoratra` $\rightarrow$ `soratra`, `mamaky` $\rightarrow$ `vaky`, `mamboly` $\rightarrow$ `voly`, `mangalatra` $\rightarrow$ `halatra`. - **Passive & imperative suffixes**: `soratana` $\rightarrow$ `soratra`, `vakina` $\rightarrow$ `vaky`, `tapaho` $\rightarrow$ `tapaka`. - **Infixes**: `vinaky` $\rightarrow$ `vaky` (infix `-in-`), `tinapaka` $\rightarrow$ `tapaka`. - **Reduplication**: `tsaratsara` $\rightarrow$ `tsara`, `moramora` $\rightarrow$ `mora`, `fotsifotsy` $\rightarrow$ `fotsy`. - **Compound words & Consonant Sandhi**: `harem-pirenena` $\rightarrow$ `harena_firenena`, `tanan-dehibe` $\rightarrow$ `tanana_lehibe`, `ara-politika` $\rightarrow$ `araka_politika`. - **Modern Malagasy Loanwords (*Teny nohagasiana*)**: `governemanta`, `politika`, `demokrasia`, `solosaina`, `telefaonina`, `banky`. - **Ultra-compact FST Dictionary**: **9,960+ pure canonical roots** compiled into a Finite State Transducer graph (~54 KB in memory, $O(k)$ lookup $ 1,500,000 words/second** per CPU core. - **Typo Tolerance**: Fast fuzzy root lookup via Levenshtein automata. - **Tokenization & Stopwords**: Smart Unicode punctuation splitter, clitic handling (`amin'ny`, …