Catalode/FIDEL-Agents

A benchmark measuring AI agent reliability on paired Amharic and English tasks in simulated Ethiopian workflows. # FIDEL Agents v1 FIDEL Agents measures whether tool-using language-model agents remain reliable when the same task is expressed in Amharic and a paired English control. It separates language-linked conditions from obstacle-isolation controls and scores completed actions, answer surfaces, and safe abstention. FIDEL Agents v1 contains 100 measured pairs and 200 task files: 24 calendar, 24 clock, 22 homophone, 12 kin-term, 4 bilingual-name, and 14 abstention pairs. Eight additional bilingual-name pairs in the private source were withdrawn and are not runnable benchmark tasks. FIDEL Agents is built by Kaleb Gebresillasie and Abhishek Mynam at Catalode Inc. See CITATION.cff for citation metadata. ## Install and validate ```sh python -m pip install -e '.[test]' python scripts/check_symmetry.py python scripts/check_aux_parity.py python scripts/smoke_test.py python -m pytest -q ``` Every command above runs without credentials or a network request. For an air-gapped setup, see the offline installation section in evaluation guidance; a wheelhouse needs `setuptools` and `wheel` alongside the declared dependencies. List providers and validate a one-pair selection without credentials or a network request: ```sh python scripts/run_evaluation.py --list-providers python scripts/run_evaluation.py --provider anthropic \ --model YOUR_MODEL --max-tasks 1 --dry-run ``` After installing the selected provider extra and supplying your own configuration: ```sh python scripts/run_evaluation.py --provider openai-chat \ --model YOUR_MODEL --tasks tasks --output episode_records ``` Supported selections are `anthropic`, `openai-chat`, `openai-responses`, `bedrock`, `gemini-api`, `vertex`, and `azure`. See evaluation guidance and the original aggregate results. The identity environment models core elements of Fayda's public OIDC and eKYC relying-party flow. Its mismatch tasks are synthetic safety stress tests, not documented production Fayda procedures. See the identity environment f …