nile-agi/nileagi-sub

Swahili Understanding Benchmark # SUB: Swahili Understanding Benchmark SUB evaluates compact open models on multiple-choice Swahili questions drawn from school-grade material. The current release compares eight models with approximately 2–5 billion parameters. ## Current results The evaluation contains 2,570 valid questions. Each model receives the same prompt and is scored by exact multiple-choice answer extraction. | Rank | Model | Parameters | Accuracy | |---:|---|---:|---:| | 1 | Gemma 4 E4B | 4.5B | **54.7%** | | 2 | AfriqueQwen3.5-4B | 4.0B | **50.3%** | | 3 | AfriqueGemma-4B | 4.0B | **43.4%** | | 4 | Tiny Aya Earth | 3.35B | **41.4%** | | 5 | Tiny Aya Global | 3.35B | **40.7%** | | 6 | Qwen3.5-4B | 4.0B | **35.4%** | | 7 | Qwen3.5-2B | 2.0B | **29.1%** | | 8 | Llama 3.2 3B | 3.2B | **27.8%** | Detailed results are in `results/results.md` and the canonical machine-readable output is `results/eval_results.json`. ## Benchmark The source dataset contains 2,608 records. The loader retains 2,570 valid multiple-choice items after rejecting malformed records and invalid answer keys. Items cover: - Standard 3–7 (`darasa_la_tatu` through `darasa_la_saba`) - Form 2–4 (`kidato_cha_pili` through `kidato_cha_nne`) - Two- to five-option questions (98.6% have four or five options) The single malformed `darasa_la_A` record is excluded from publication plots. ## Installation Python 3.11 and a CUDA-capable GPU are recommended for local Hugging Face models. ```bash python3 -m venv .venv source .venv/bin/activate pip install -r requirements.txt cp .env.example .env ``` Set `COHERE_API_KEY` for Tiny Aya. Set `HF_TOKEN` when access to a gated Hugging Face model requires authentication. ## Evaluate List configured models without loading them: ```bash python src/eval.py --list ``` Evaluate all models: ```bash python src/eval.py ``` Evaluate selected models: ```bash python src/eval.py --models qwen3.5-2b qwen3.5-4b ``` The evaluator: - skips model rows already present in the output unless `--f …