umair-a11y/polyglot-eval

Eval harness that flags LLM quality regression in low resource languages. Refuses to emit scores its judge cannot trust. # polyglot-eval Run an LLM eval suite across multiple locales and flag quality regressions in low-resource languages that English-first eval tools tend to miss. Status: v0.1. Working and tested offline. The real API backends are wired but have only been exercised by hand, not by the test suite (see Limitations). ## The problem Most eval tools (promptfoo, deepeval, and friends) are excellent, but their default rubrics, examples, and intuitions are built around English. When you ship a product that also serves Hindi, Urdu, Punjabi, or Arabic speakers, an English-passing eval can stay green while the experience in those languages quietly degrades. The model writes stilted, literal, or subtly wrong text and nobody notices, because the suite never compared per language. polyglot-eval makes the per-language comparison the main thing. It runs every test case across every locale you list, scores each output with a language-aware LLM judge, and tells you which locales fell behind your English baseline by more than a threshold you set. The maintainer is a native Urdu, Hindi, and Punjabi speaker. That is why this tool leans hard on one honest admission: an LLM judging output in a low-resource language is itself unreliable. So every judge score carries a confidence, and low-confidence scores get a "needs-human" label instead of a made-up number. ## Quick start (offline, no API key) You need Bun. Then: ```sh bun install bun run src/cli.ts run examples/evalset.yaml ``` This runs the bundled example with deterministic MOCK backends. No key, no network. It produces a real report immediately. The mock judge deliberately simulates the effect this tool exists to catch: lower quality and lower judge confidence in low-resource locales. Example output: ``` polyglot-eval report ============================================================ baseline=en threshold=0.15 confidence-floor=0.5 Per-locale quality (trusted mean; needs-human excluded) ---------------------------------- …