sudaverse/sudaverse-normalizer

Sudanese Arabic text normalization and cleaning toolkit # Sudaverse Normalizer 🇸🇩 A robust, production-ready text normalizer specifically designed for **Sudanese Arabic dialect**. This tool handles the unique characteristics of Sudanese Arabic, including dialect-specific spelling variations, colloquialisms, and mixed Arabic-Latin text. ## ✨ Features - **🎯 Sudanese Dialect-Specific**: Tailored for Sudanese Arabic text patterns and common expressions - **🔤 Unicode Normalization**: Proper handling of Arabic text encoding (NFKC, NFC, NFD, NFKD) - **📝 Diacritic Handling**: Remove or preserve Arabic diacritics (tashkeel) with flexible options - **🔢 Number Normalization**: Convert Arabic-Indic numerals to Western numerals - **✂️ Character Normalization**: - Normalize all Alef variants (أ، إ، آ، ٱ) to ا - Normalize Yeh variants (ى، ئ) to ي - Normalize Teh Marbuta (ة) to Heh (ه) - **🧹 Cleaning Features**: - Remove URLs, emails, mentions, and hashtags - Remove HTML/XML tags and unrecognized special characters - Remove decorative lines made of tatweel/kashida characters (ـــــــــــ) - Remove English/Latin text and timestamps - Remove repeated characters (e.g., "كتييييير" → "كتيير") - Normalize punctuation and whitespace - **⚙️ Highly Configurable**: 20+ configuration options for custom normalization pipelines - **📊 Batch Processing**: Efficient folder-based processing with real-time progress tracking - **📈 Statistics**: Get detailed statistics about text transformation ## 📦 Installation ### Requirements - Python 3.7 or higher ### Install Dependencies ```bash pip install -r requirements.txt ``` ## 🚀 Quick Start Guide ### For First-Time Users #### Step 1: Prepare Your Text Files 1. Open the `raw-text` folder in your project directory 2. Place your Sudanese Arabic text files (`.txt` format) in this folder 3. You can have as many files as you want! #### Step 2: Run the Batch Processor Open your terminal in the project directory and run: ```bash python batch_processor.py ``` #### Step 3: Get Your Results - ✅ Normaliz …