NickDee96/kikuyu-cleaner

A standalone Python library for cleaning and normalizing Kikuyu (Gĩkũyũ) text using frequency-based analysis and pattern matching. # Kikuyu Text Cleaner A standalone Python library for cleaning and normalizing Kikuyu (Gĩkũyũ) text. This package operates **without any external dictionary dependencies**, using frequency-based analysis and pattern matching for text standardization. ## Features - ✅ **Unicode normalization** and standardization (NFC/NFD) - ✅ **Diacritic detection and correction** (ĩ/i, ũ/u variations) - ✅ **Transcript marker cleaning** (removes `[Pause]`, `[cs]`, etc.) - ✅ **Spelling variant detection** based on frequency analysis - ✅ **Context-based validation** of spelling variants - ✅ **Standard Kikuyu pattern corrections** (common function words) - ✅ **Case preservation** during corrections - ✅ **Comprehensive analysis reports** ## Installation The package is contained within the `kikuyu_cleaner` folder. No additional installation is required beyond having Python 3.8+ and optionally `rapidfuzz` for edit distance calculations. ```bash # Optional: Install rapidfuzz for enhanced edit distance analysis pip install rapidfuzz ``` ## Quick Start ### Python API ```python from kikuyu_cleaner import KikuyuCleaner, KikuyuAnalyzer # === Basic Cleaning === texts = [ "Ni mundu muno na andu acio", "Tondu undu uria ni mwega", "[Pause] Ati mari na uhoro [cs]", ] # Create cleaner and build correction maps cleaner = KikuyuCleaner(texts) cleaner.build_correction_maps() # Process texts corrected_texts, changes, stats = cleaner.process_texts(texts, clean_first=True) print(f"Modified {stats['rows_modified']} rows") print(f"Applied {stats['total_corrections']} corrections") # === Analysis Only === analyzer = KikuyuAnalyzer(texts) results = analyzer.run_full_analysis() print(f"Found {len(results['diacritic_variants'])} diacritic variant groups") print(f"Found {len(results['diacritic_confusion_pairs'])} confusion pairs") ``` ### Command Line Interface ```bash # Clean a CSV file python -m kikuyu_cleaner.cli clean input.csv --column text --output cleaned.csv # Analyze texts python -m kiku …