gamzegulmungan/geoai-reliability-audit-abuja

Instance-level reliability audit of a deployed GeoAI building-extraction system (Abuja, Nigeria) — recall, IoU, sensitivity analysis, ground-truthed failure modes # GeoAI Reliability Audit — Abuja, Nigeria **An instance-level empirical reliability audit of AI-generated building footprints in a data-scarce urban environment.** This repository accompanies the technical report *"Quantifying the Reliability Gap: A Spatial Accuracy Assessment of AI-Generated Building Footprints in a Data-Scarce Urban Environment (Abuja, Nigeria)"* (G. G. Mungan, 2026) — prepared as a technical work sample for an AI safety fellowship application. Full report: `Abuja_GeoAI_Reliability_Report_FINAL.pdf`. ## The question AI systems increasingly generate building-footprint data where OpenStreetMap coverage is incomplete, and that data feeds digital twins, disaster-risk mapping, and infrastructure planning. **How reliable is each individual AI-generated geometry — not just the aggregate volume?** ## Key findings (4.28 km² defined study area, Abuja) | Metric | Value | |---|---| | OSM reference buildings | 94 | | AI-generated buildings (Mapflow.ai) | 684 (**7.3×** the OSM count) | | Recall vs OSM baseline (IoU ≥ 0.1) | **56.4%** — 41 known buildings undetected | | Recall at strict COCO-style IoU ≥ 0.5 | **25.5%** | | Mean IoU among confirmed matches | 0.458 (median 0.425) | | Mean centroid offset | **6.03 m** (max 14.0 m) | | AI-only detections (unverified) | 91.8% of AI output | **Core safety-relevant finding:** a system that generates 7.3× more building polygons than the existing baseline *simultaneously* fails to detect over 40% of independently known buildings. Aggregate coverage and per-instance reliability are distinct properties — evidence of one is not evidence of the other. A ground-truthed failure case (stacked concrete pipes labeled as buildings, confirmed via street-level imagery) further shows that overhead-imagery-only verification is itself insufficient for oversight design. ## Repository structure This repository uses a flat layout — all files sit in the root: ``` matching_analysis.py # full pipeline: load → …