Ey0s/amharic_nlp_toolkit

Lightweight Amharic NLP toolkit for normalization, tokenization, stopword filtering, stemming, and basic linguistic tagging. # amnlp Lightweight Amharic NLP toolkit for normalization, tokenization, stopword filtering, stemming, and basic linguistic tagging. ## Table of Contents - Overview - System Architecture - Project Structure - Installation - Quick Start - Core API - Data and Resource Pipelines - Testing - Design Notes and Limitations - Roadmap Ideas ## Overview `amnlp` provides rule-based building blocks for processing Amharic text. Current capabilities: - Ethiopic-script tokenization - Orthographic normalization and punctuation cleanup - Prefix splitting for common Amharic prefixes - Stopword removal (file-backed list) - Suffix-based stemming - Basic morphology, POS, and NER heuristics - Dictionary-backed word existence checks - Composable mini pipeline abstraction ## System Architecture ### High-level processing flow ```text Raw Text | v AmharicNormalizer.normalize | v AmharicTokenizer.tokenize | v split_tokens (prefix splitter) | v StopwordRemover.remove | v AmharicStemmer.stem | v Deduplicate (order-preserving) | v Processed token list ``` ### Component architecture | Layer | Module | Class/Function | Responsibility | |---|---|---|---| | Public API | `amnlp/__init__.py` | `tokenize`, `stem`, `remove_stopwords` | Simple helper entry points | | Pipeline | `amnlp/pipeline/processor.py` | `AmharicProcessor` | End-to-end text processing pipeline | | Pipeline Infra | `amnlp/pipeline/pipeline.py` | `Pipeline` | Generic component chaining (`add`, `run`) | | Tokenization | `amnlp/tokenizer/tokenizer.py` | `AmharicTokenizer` | Extract Ethiopic word tokens via regex | | Sentence Split | `amnlp/tokenizer/sentence_tokenizer.py` | `SentenceTokenizer` | Split by sentence punctuation | | Normalization | `amnlp/normalization/normalizer.py` | `AmharicNormalizer` | Character substitutions + punctuation removal | | Stopwords | `amnlp/stopwords/stopwords.py` | `StopwordRemover` | Remove tokens from resource stopword list | | Stemming | `amnlp/stemmer/stemmer.py` | `AmharicStemmer` | Rule-ba …