makda-tsegazeab/tigrinya-nlp

# Tigrinya NLP Toolkit Tigrinya NLP Toolkit is a lightweight, practical, and easy-to-use preprocessing library for Tigrinya text (Ethiopic/Ge'ez script). It provides safe, transparent utilities for cleaning, normalization, tokenization, and stopword removal. Perfect for research, machine learning, and NLP pipelines that need reliable Tigrinya preprocessing without aggressive or irreversible changes. ## 🌍 Why Tigrinya Needs Its Own NLP Toolkit Tigrinya is morphologically rich and written in Ethiopic script. General-purpose NLP tools often struggle with: - Ethiopic punctuation and spacing - Unicode normalization inconsistencies - Script-specific word boundaries - Mixed-script or noisy social text This toolkit provides a conservative, language-aware preprocessing pipeline built specifically for Tigrinya. ## ⚙️ What Is tigrinya-nlp? tigrinya-nlp is a modular Python package for end-to-end Tigrinya preprocessing. ### 🧩 Core Components - Cleaner: removes URLs, emojis, mentions, hashtags, and repeated punctuation (configurable) - Normalizer: Unicode NFC, invisible character removal, punctuation spacing, whitespace fixes - Tokenizer: sentence and word tokenization with Ethiopic-aware punctuation rules - Stopword Processor: curated stopword lists with configurable categories ### ✅ Intentionally Out of Scope (for now) - Stemming - Lemmatization - Spell correction - Morphological analysis ## 📦 Installation Option 1: Install from PyPI (Recommended) ```bash pip install tigrinya-nlp ``` Option 2: Install Latest Development Version ```bash git clone github.com cd tigrinya-nlp pip install . ``` ## 🧪 Full Demo: End-to-End Tigrinya Text Preprocessing ```python from tigrinya_nlp import clean, normalize, words, remove_stopwords sample_text = "ዝተረፈ ጽሑፍ ብቕልጡፍ ንኣብዚ መለኪያ example.com" # Step 1: Cleaning cleaned = clean(sample_text) # Step 2: Normalization (conservative) normalized = normalize(cleaned) # Step 3: Tokenization …