okech-christopher/sheng-sentiment-base

High-fidelity NLP infrastructure for Sheng and East African code-switching. Building the 'Cultural API' for the informal economy. # Sheng-Sentiment-Base **Project Sheng V1**: The foundational NLP infrastructure for Kenya's informal economy. > *"Building the engine, not just the car."* ## Overview Sheng-Sentiment-Base is a high-fidelity NLP preprocessing engine for **Sheng**—Nairobi's dynamic urban slang that blends Swahili, English, and local dialects. This infrastructure layer enables accurate sentiment analysis, text classification, and language understanding where generic LLMs fail. ### The Problem Global AI models (GPT-4, Gemini) struggle with: - **Code-switching patterns**: "Hii mbogi ni fiti" → "This crew is good" - **Rapidly evolving slang**: New terms emerge weekly in Nairobi's streets - **Context-dependent sentiment**: "Kudunda" can mean partying (positive) or failing (negative) ### The Solution A specialized "Cultural API" that: - Normalizes Sheng variant spellings ('ronga' → 'rongai') - Detects code-switching boundaries - Applies contextual sentiment rules - Provides training-ready datasets for fine-tuning ## Quick Start ```bash # Clone the repository git clone github.com cd sheng-sentiment-base # Install dependencies pip install -r requirements.txt # Run the tokenizer python -m src.tokenizers.sheng_tokenizer ``` ## Usage ### API Usage (FastAPI Service) Start the API server: ```bash # Development mode python -m src.api.main # Production mode uvicorn src.api.main:app --host 0.0.0.0 --port 8000 --workers 4 ``` **Analyze Sheng text via API:** ```bash curl -X POST localhost \ -H "Content-Type: application/json" \ -d '{ "text": "Karao wako mabs, jam imetupa", "include_logistics": true, "include_code_switches": true }' ``` **Response:** ```json { "original_text": "Karao wako mabs, jam imetupa", "normalized_text": "karao wako mabs jam imetupa", "tokens": ["karao", "wako", "mabs", "jam", "imetupa"], "slang_terms": ["karao", "mabs", "jam"], "code_switches": [], "sentiment_score": -0.5, "sentiment …