AnyEvalOrg/eval-uccb

Uganda Cultural and Cognitive Benchmark (UCCB) — packaged for OpenEvalz (plain) # eval-uccb **Uganda Cultural and Cognitive Benchmark (UCCB)** **Paper:** Ugandan Cultural Context Be… The first comprehensive question-answering dataset designed to evaluate cultural understanding and reasoning abilities of Large Language Models concerning Uganda's multifaceted environment across 24 cultural domains including education, traditional medicine, media, economy, literature, and social norms. ## At a glance | | | |---|---| | Upstream | `src/inspect_evals/uccb` | | Group | Knowledge | | Total samples | 1,039 | | Execution class | `plain` | | Cost class | `medium` | | Flags | no sandbox, no network | | Tags | — | ### Tasks | Task | Samples | |---|---| | `uccb` | 1,039 | ### External assets - `huggingface` — `CraneAILabs/UCCB` (pinned) ## Running one problem OpenEvalz is problem-level: the atomic unit is a single sample, not the whole eval. ```bash inspect eval inspect_evals/uccb \ --sample-id " " \ --model openai-api/trustedrouter/ \ --token-limit 200000 ``` > **Two things that bite here, both verified in Inspect's source.** > > 1. **`--cost-limit` does not work on this routing path.** Inspect only records cost when its > pricing table resolves the model, and `_model_info.py` strips only `azure|bedrock|vertex` > prefixes — so `trustedrouter/ ` never resolves and the cap silently never binds. The > real spend cap is enforced **server-side by TrustedRouter** via the delegated key's > `limit_microdollars` and spend window. Use `--token-limit` as the in-process bound. > 2. **`--sample-id` matches with `fnmatch`.** A glob silently selects many samples and only warns. > Always pass a literal id. ## Reproducibility `bundle.template.json` is the contract. A run that cannot emit a complete bundle does not publish. Every image is pinned by `sha256` digest and every dataset by revision. ## Licensing OpenEvalz wrapper code in this repository is **Business Source License 1.1** (see `LICENSE`) — Licensor Lore Hex Corp …