KennOmollo-Ke/LID

A lightweight machine learning system that detects the language of short text inputs in English, Swahili, Sheng, and Dholuo using TF-IDF and classification models. # LID – Language Identification System A lightweight machine learning system that detects the language of short text inputs in **English, Swahili, Sheng, and Dholuo**. Built for real-world Kenyan digital communication where messages are short, informal, and often multilingual. --- ## Features - Detects language from short text (1–2 sentences) - Supports English, Swahili, Sheng, and Dholuo - Uses TF-IDF with character n-grams - High accuracy (~95%) - Real-time predictions via Streamlit app - Confidence score for each prediction --- ## Dataset Summary The dataset was manually curated to reflect real-world communication patterns. - **Total Samples:** 3,021 ### Language Distribution | Language | Samples | | -------- | ------- | | English | 991 | | Swahili | 880 | | Sheng | 717 | | Dholuo | 433 | The dataset consists of short, informal text collected from social media, SMS-style messages, and conversational phrases. --- ## 🧠 How the Model Works 1. **Preprocessing** - Lowercasing - Noise removal - Cleaning informal text 2. **Feature Extraction** - TF-IDF vectorization - Character n-grams (2–4) 3. **Models Used** - Naive Bayes - Logistic Regression (Best Model) 4. **Evaluation Metrics** - Accuracy - Precision - Recall - F1-score --- ## 📁 Project Structure ```text LID/ ├── app.py ├── train_model.py ├── requirements.txt ├── README.md ├── data/ ├── utils/ ## How to Run pip install -r requirements.txt 3. Start the app: streamlit run app.py ## Re-Train the Model python -m pip install --upgrade pip python -m pip install pandas scikit-learn streamlit matplotlib python -c "import sys; print(sys.executable)" python train_model.py ```