nd-hal/swahili_psych_taxonomy

# Benchmarking Sociolinguistic Diversity in Swahili NLP: A Taxonomy-Guided Approach This repository accompanies the paper: **Add PDF of the paper** --- ## Overview We introduce the first taxonomy-guided evaluation of Swahili NLP using a new dataset of 2,170 free-text responses collected from Kenyan speakers in response to health-related psychometric tasks. The dataset captures rich sociolinguistic variation in real-world Swahili, including: - Sheng (urban Swahili-English hybrid) - Tribal lexicons - Code-mixing - Loanwords We benchmark both pre-trained and instruction-tuned models (e.g., mBERT, XLM-RoBERTa, SwahBERT, Llama, Qwen) and analyze fairness and performance across demographic and linguistic aspects. --- # Repository Structure ``` 📂 Data/ # Contains the Swahili dataset and model predictions 📂 Results/ # Folder to store outputs 📂 Scripts/ # Python and R script(s) for model evaluation and plot generation 📜 README.md # This file 📜 poetry.lock # Dependency lock file for reproducibility 📜 pyproject.toml # Configuration for managing dependencies with Poetry ``` ## Dataset Summary - 2,170 open-ended Swahili text responses - Four psychometric tasks: Anxiety, Trust, Literacy, Numeracy - Annotated with demographic metadata: tribe, gender, age, education, income - Includes non-standard languistic features: Sheng, loanwords, dialectal Swahili, and code-mixing --- ## Tasks and Benchmarks We evaluate a range of multilingual pre-trained language and instruction-tuned models using both regression and classification tasks. Metrics include: - Pearson correlation and RMSE for regression - AUC and F1 for classification - Group fairness metrics such as Disparate Impact, ∆xAUC, and Fairness Violation --- # Setup & Installation This project uses **Poetry** for dependency management. 1. Download pipx: pipx.pypa.io 2. Install poetry: python-poetry.org 3. To …