gilbert215/Language-ID

NLP classifier identifying 6 Latin-script languages (Hausa, Indonesian, Manobo, Nahuatl, Swahili, Tagalog) using character n-grams and Multinomial Logistic Regression. Leverages character frequency patterns and phonotactics for 85-95% accuracy. Solves multinomial classification with feature engineering. Python, scikit-learn. # Language Identification Classifier ## 📋 Project Description A character-based machine learning classifier that automatically identifies which of six languages a text is written in: Hausa, Indonesian, Manobo, Nahuatl, Swahili, and Tagalog. Since all six languages use the Latin script with minimal diacritics, making visual distinction difficult, the system leverages character frequency patterns and phonotactic structures (how sounds combine) using n-gram features and Multinomial Logistic Regression (Softmax Regression) to achieve accurate language detection. --- ## 🎯 The Challenge **Problem:** Distinguish between six languages that look very similar: **The Six Languages:** 1. **Hausa** - West African language 2. **Indonesian** - Southeast Asian language 3. **Manobo** - Philippine indigenous language 4. **Nahuatl** - Mexican indigenous language 5. **Swahili** - East African language 6. **Tagalog** - Philippine national language **Why It's Hard:** - All use the **same Latin alphabet** - Minimal use of special characters (diacritics) - Without linguistic training, humans can't easily tell them apart - Can't rely on unique characters for each language **Why Traditional Approaches Don't Work:** - **Word-based features:** Too many possible words = sparse, overwhelming feature space - **Simple pattern matching:** No unique character sets to distinguish languages - **Binary classification:** This is a 6-way (multinomial) classification problem --- ## 💡 Solution Approach ### Why Character-Based Features? **Key Insight:** Languages have unique character frequency patterns and sound combination rules (phonotactics). **Two Feature Types:** 1. **Character Unigrams (Single Characters)** - Each language uses letters with different frequencies - Example: 'q' common in some languages, rare in others - Captures overall character distribution 2. **Character N-grams (Character Sequences)** - Bigrams: 2-character sequences ("th", "ng", "ua") - Trigrams: 3-character sequen …