gabrielmahia/kipimo

Swahili agent-task evaluation suite — model-agnostic seed benchmark for the East Africa coordination stack (46 tasks; golds machine-derived from authoritative sources). # kipimo Over one hundred million people coordinate their lives in Swahili, yet no benchmark measures whether an AI agent can route their requests correctly — send money, check drought status, find a clinic, verify a worker's credentials. Agents targeting East Africa are evaluated on English tasks and deployed on faith. `kipimo` (Swahili: *a measure*) is a model-agnostic seed benchmark for exactly that gap: **46 tasks** across three types, with golds machine-derived from authoritative sources — the coordination-stack registry and the live `africa-coord-bus` routing table — never from memory. | Type | n | What it measures | Metric | |---|---|---|---| | `server_routing` | 25 | Swahili request → correct stack server (payments, tax, health, land, labour…) | exact | | `term_grounding` | 14 | Swahili domain term → English meaning | exact (case-insensitive) | | `cascade_routing` | 7 | Coordination event → which sectors must be notified | set F1 | **Live leaderboard →** — score any model in your browser, no keys. ## Use it (any model, no API keys) ```bash pip install kipimo kipimo tasks > tasks.jsonl # feed to your agent however you like kipimo template > preds.jsonl # fill "prediction": [...] per id kipimo score preds.jsonl # per-type + overall report ``` The harness never calls a model — you generate predictions with whatever system you're evaluating; kipimo only scores. Any lab can publish comparable numbers. ## Data sovereignty The models deployable under African data-sovereignty constraints are open-weight and self-hostable. kipimo scores that tier on equal terms with frontier APIs so builders can test whether the model they are *allowed* to run is good enough — see docs/DATA_SOVEREIGNTY.md and `examples/generate_predictions.py`. ## Honesty box - **v0.1 is a seed set.** 46 tasks establish the format and scoring; breadth comes from contributions. - Swahili phrasing is simple-register and **pending native-speaker review** — that is issue #1, …