Eng-Muscab/SOMALI-NLP-RESEARCH

# Somali NLP Research This project is based on the labeled binary dataset in: - `data/raw/labeled text.xlsx` The active task is **AI vs HUMAN Somali text classification**. The larger `data/raw/full_dataset.xlsx` is kept as supplemental/reference data for EDA/category analytics, but the supervised labels used for model training come from `labeled text.xlsx`. ## Experiments The two canonical experiment folders are: - `experiments/experiment_1_stopwords_included/` - `experiments/experiment_2_stopwords_removed/` Each experiment contains its own data splits, trained models, evaluation reports, figures, XAI outputs, and final result summaries. ## Installation and Setup (For Cloning the Repository) If you are cloning this repository for the first time, follow these steps to set up the environment and download the trained models: ### 1. Install Git LFS (Large File Storage) The trained model weights (e.g., `.safetensors`, `.keras`, `.joblib`, `.model`) are stored using Git LFS. - Install Git LFS on your system (if not already installed). - Run the following command once in your terminal: ```bash git lfs install ``` - Clone the repository: ```bash git clone cd somali-nlp-research ``` - If the model files did not download automatically during cloning, run the following to pull them: ```bash git lfs pull ``` ### 2. Create a Virtual Environment It is highly recommended to use a local virtual environment: ```bash # Create the environment python -m venv .venv # Activate it: # On Windows (PowerShell): .venv\Scripts\Activate.ps1 # On macOS/Linux: source .venv/bin/activate ``` ### 3. Install Dependencies Install all required Python libraries: ```bash pip install -r requirements.txt ``` ### 4. Verify Model Setup You can run the model evaluation script to verify that the models have loaded and run correctly: ```bash python experiments/reevaluate_saved_models.py ``` ## Dataset After cleaning, conflict removal, and duplicate removal: | Label | Rows | |---|---:| | AI | 5 …