LLM Reporting of Official Statistics: Benchmarking the Accuracy and Reliability of Six Commercial LLMs from Frontier AI Labs Across 126 UNICEF-Published Child Indicators

This study benchmarks two complementary dimensions of large-language-model (LLM) behaviour when asked, in **natural language**, to report official child statistics: (1) **accuracy** — when a commercial LLM from a frontier AI lab is queried by indicator name (e.g., "under-five mortality rate") for a country (e.g., "Burkina Faso") and a year for which an official UNICEF value exists, how often does it correctly retrieve and report the value? — and (2) **reliability** — when the same natural-language query is issued twice across separate benchmark runs under identical model-snapshot, prompt-template, and ground-truth conditions, how consistent are the LLM's answers? Both are evaluated alongside a third probe of (3) **appropriate refusal under impossible queries**, which tests whether models hallucinate plausible values for statistics that do not exist in the official record. A fourth, cross-cutting objective is (4) **heterogeneity** — each of the three results above is decomposed across **UNICEF regions**, **World Bank country income groupings**, and **indicator domains** (SDG goal), testing whether the headline accuracy, reliability, and refusal behaviour vary systematically by region, income, or domain. Six widely-deployed commercial LLMs from frontier AI labs (two each from Anthropic, OpenAI, and Google, spanning flagship and efficiency tiers) are queried with six prompt variants across 3,711 (indicator × country × year) triplets accessed via the UNICEF SDMX API (plus the ILO SDMX API for the two NEET indicators, where UNICEF SDMX does not host the canonical series), producing 133,596 model responses on the has-data stratum. The six prompts are five natural-language phrasings (`baseline_latest`, `direct`, `concise`, `json_structured`, `chain_of_thought`) and one SDMX-code variant (`direct_code`) included to test the code-vs-natural-language framing question on a triplet-matched paired contrast against `direct` (RQ11). Each response is scored against a freshly fetched UNICEF ground-truth value via a deterministic layered extractor and the result is summarised by the **Expected Query Accuracy (EQA)** metric — the primary outcome of the study. UNICEF SDMX is the API and metadata channel by which official statistics are accessed; it is infrastructure rather than the substantive object of the study, and the five natural-language prompts use indicator and country _names_ (not SDMX codes), per the study's natural-language intent; the sixth `direct_code` prompt is the deliberate SDMX-code counterfactual against which the natural-language framing is benchmarked. **Primary outcome — Expected Query Accuracy (EQA)**. EQA is a multiplicative composite of three components scored per LLM response: `EQA = ER × YA × VA` - **ER (Extraction Rate)** — did the LLM produce a numeric value at all, after refusal-filtering and abstain-gating? Output of the deterministic layered extractor at canonical version **`det-v5.3+v3.3+v2.3`** (entry point `extract_with_layered_info_v5` in [`src/unicef_llm_benchmark/evaluation/numeric_accuracy_v5.py`](../src/unicef_llm_benchmark/evaluation/numeric_accuracy_v5.py); architecture is NER → RFER → ER stack with the V5.3 abstain layer and four Phase-1A discriminator gates calibrated against a 272-case cross-vendor three-judge panel: Claude Sonnet 4.5, GPT-4o, Gemini 2.5 Flash). The two extractor source modules (`numeric_accuracy_v5.py` and `numeric_accuracy.py`, the latter housing the V1 / V2.3 / V3.3 layers that V5.3 wraps) are sha256-pinned in `config/v9/lockfile.json` so the implementation is auditable byte-for-byte at replication time. - **YA (Year Accuracy)** — does the year reported in the response match the queried year? - **VA (Value Accuracy)** — does the extracted value fall within the indicator-specific precision tolerance of the official UNICEF value? The multiplicative structure means partial success at any one stage propagates: a response that extracts a value but for the wrong year (YA = 0) or outside the precision tolerance (VA = 0) scores EQA = 0 even if ER = 1. EQA is therefore the more honest benchmark for policy use, since a policymaker needs _the right number for the right year at the right precision_, not just any plausible-looking number. The three components are each reported as **secondary outcomes** so that EQA failures are fully decomposable: when an LLM scores EQA = 0 on a query, the ER / YA / VA decomposition tells reviewers whether the failure was extraction (model produced nothing or refused), temporal anchoring (right value, wrong year), or precision (right concept, value outside tolerance). This decomposability is a deliberate design strength — it converts a black-box accuracy number into a diagnostic about where LLMs fail. The headline estimands are: 1. **Accuracy headline** — pooled EQA across the 3,711 × 6 × 6 factorial design (six models × six prompts). 2. **Reliability headline** — mean within-pair ΔEQA across a matched-pair second benchmark run on a 99-triplet subset, testing whether identical natural-language queries produce consistent answers across runs. 3. **Hallucination headline** — falsification proportion per (model × prompt) cell on a 500-triplet expected-no-data set (250 future-year impossibilities uniformly drawn from 2027–2050 + 250 historical gaps within 2000–2025; see the "Study design" answer below for the sub-stratum specification), testing whether models fabricate values when no data exists. 4. **Framing-choice contrast (RQ11)** — paired within-triplet ΔEQA between the `direct_code` SDMX-code prompt and the `direct` natural-language prompt, per model, testing whether the choice of code-vs-natural-language framing materially changes EQA when all other factors are held constant. 5. **Heterogeneity stratification** — each of headlines (1)–(3) is also reported stratified by (a) **`strata_region`** (7 substantive levels: Eastern and Southern Africa, West and Central Africa, Latin America and the Caribbean, Northern Africa and Western Asia, Eastern and South-Eastern Asia, Central and Southern Asia, plus a merged "Oceania, Europe and Northern America" stratum), (b) **World Bank income group** (4 substantive levels: LIC, LMC, UMC, HIC), and (c) **indicator domain** (SDG goal, restricted to the 19 UNICEF co-custodian SDGs). The corresponding interaction tests (RQ3) test whether observed model and prompt effects differ systematically by region or income; the per-SDG breakdown (RQ5) tests whether they differ by domain. Pre-specified disclosure rule: any stratified result whose 95% CI excludes the pooled headline by more than the headline CI half-width is reported as a substantive heterogeneity finding in the manuscript. The estimand denominator for the accuracy headline is the population of official child statistics that exist in the UNICEF data warehouse — not the world's child population. A population-extrapolated estimate accompanies the accuracy headline as a robustness check that documents which children's statistics are over- or under-represented in the sample. The reliability headline is paired by construction and does not require population weighting. The 126-indicator benchmark is a stratified-purposive (mixed) sample combining 75 composite-ranked data-rich indicators with 51 mandate-selected indicators chosen to close UNICEF co-custodian SDG-indicator coverage to 19 of 19 and the SDGnetAnalysis "Progress for Children" indicator set to ~96%. Within each indicator, country-year triplets are drawn by stratified random sampling across **`strata_region` (7 levels: 4 Global South UNSD M49 SDG regions — Latin America and the Caribbean, Northern Africa and Western Asia, Eastern and South-Eastern Asia, Central and Southern Asia — plus the UNICEF programmatic split of Sub-Saharan Africa into Eastern and Southern Africa + West and Central Africa, plus 1 merged "Oceania, Europe and Northern America" stratum aggregating the canonical UNSD `Europe and Northern America` + `Oceania (excluding Australia and New Zealand)` + `Australia and New Zealand`) × World Bank income group (4 substantive levels: LIC, LMC, UMC, HIC)**. The Sub-Saharan Africa split follows UNICEF's programmatic taxonomy; for ESA + WCA the assignment uses `unicef_region` rather than `sdg_region` so border cases (e.g., Djibouti, programmatically in ESA but UNSD-classified in Northern Africa and Western Asia) follow the UNICEF split. The country metadata layer (`config/v9/country_metadata.csv`) preserves the canonical UNSD `sdg_region` and UNICEF programmatic `unicef_region` columns untouched as lineage and adds a separate `analytic_region` column carrying the M49 5-continent macro classification (Africa, Americas, Asia, Europe, Oceania) for cross-continental reporting. **All ground-truth values, LLM responses, and extractor outputs are collected fresh at the study build date.** This benchmark is a single fresh-draw design: no triplets, prompts, responses, or extractor outputs are carried over from any earlier benchmark vintage.