ethiopicai/Amharic-Text-Processor

Modular Amharic text preprocessing toolkit with composable processors and pipeline. # Amharic Text Processor **Amharic Text Processor** is a modular Python toolkit for cleaning, normalizing, and formatting Amharic text. Each processing step is a small class with a predictable `.apply()` method, and steps are easily chained with `Pipeline`. **Why this exists:** Amharic text from the web, documents, and OCR often arrives with HTML noise, mixed Ethiopic variants, inconsistent punctuation, legacy abbreviations, and numerals in different forms. This toolkit provides predictable, composable processors so you can rapidly build robust pipelines for ML datasets, search indexing, or downstream NLP tasks without reinventing cleaning logic. Many of these components were developed while processing large volumes of Amharic text crawled from Amharic-focused websites indexed by Common Crawl. --- ## ✨ Features - Composable pipeline built from simple processor classes - Consistent I/O contract: accepts `str` or `{"text": ...}`, returns a dict with `"text"` - HTML stripping, whitespace cleanup, Amharic character filtering - Punctuation and Unicode normalization (keeps Ethiopic marks, preserves decimals) plus configurable regex filtering - Sentence-level deduplication using fuzzy similarity - Abbreviation handling for slash/dot forms; dotted abbreviations can be normalized before expansion - Helpers to add spaces between Ethiopic letters and digits, and to place sentences on separate lines - Noise removal for common Latin/underscore tokens and foreign-only brackets - Ethiopic→Latin transliteration using a romanization table - Pure, side-effect-free processors that are easy to test and extend --- ## 📦 Installation ```bash pip install amharic-text-processor ``` --- ## 🚀 Quick Start ```python from amharic_text_processor import Pipeline from amharic_text_processor.processors import ( HtmlStripper, WhitespaceNormalizer, PunctuationNormalizer, UnicodeNormalizer, CharacterRemapper, AbbreviationExpander, DottedAbbreviationNormalizer, AmharicCharacterFilter, Common …