kigalithm/rwalang

An enhanced language detector for Kinyarwanda, designed to accurately identify Kinyarwanda text, including code-mixed content involving English, French, and Swahili. # kgt-rwalang An enhanced language detector for Kinyarwanda, designed to accurately identify Kinyarwanda text, including code-mixed content involving English, French, and Swahili. > **Note:** It is not perfect yet, but it is the best there is, and it works. ## Overview kgt-rwalang combines traditional methods like character n-grams and TF-IDF with Kinyarwanda-specific linguistic features to achieve robust language detection. It includes specialized handling and confidence scoring for text that mixes Kinyarwanda with common foreign loan words and grammatical structures. ## Features * **Character N-grams & TF-IDF:** Standard text feature extraction. * **Kinyarwanda Linguistic Features:** Incorporates analysis of ibihekane, ibyungo, accented vowels, grammatical markers, and common affixes. * **Code-Mixing Detection:** Includes logic to identify and better handle text containing a mix of Kinyarwanda and foreign language elements (loan words, grammatical patterns). * **Ensemble Model:** Uses a combination of machine learning classifiers for improved accuracy. * **Model Persistence:** Ability to save and load trained models using `joblib`. * **Configurable Thresholds:** Adjust detection sensitivity. ## Installation You can install the package using pip: ```bash pip install kgt-rwalang ``` ## Usage Here's a basic example of how to use the `KinyaLangDetector` class: ```python import pandas as pd from rwalang.detector import KinyaLangDetector # Instantiate the Detector detector = KinyaLangDetector() # Load or Train the Model --- try: # Attempt to load the model. # Call load_model as is or pass a path to your own model. detector.load_model() except FileNotFoundError: print(f"Model file not found. Training new model...") try: training_df = pd.read_csv('path/to/your_training_data.csv', encoding='utf-8') except FileNotFoundError: print("Error: Training data CSV not found. Cannot train model.") # Handle this error - maybe exit or get data another way training_df = …