Lucas-Maingi/mizani

East African economic & mobile-money data platform - medallion ETL with Dagster, dbt, DuckDB, and quarantine-first data quality. Live weekly pipeline publishing to GitHub Pages. # Mizani — East African economic & mobile-money data platform **Medallion ETL · Dagster · dbt · DuckDB · data quality** — a bronze/silver/gold pipeline that ingests four genuinely messy public data sources (one JSON API, one headerless CSV, one HTML scrape, one human-formatted Excel workbook), validates and quarantines them declaratively, and models them into a star schema. *Mizani* is Swahili for "scales / balance." ## Data sources All sources were verified live on 2026-07-15. Every number below is measured from an actual pipeline run on that date, not estimated. | Source | Format | The mess | Rows landed | |---|---|---|---| | World Bank Open Data API | JSON API | nulls for missing years; pagination | 208 | | CBK historical forex rates | CSV | **no header row**, 485 exact duplicate rows (1.3%), ambiguous MM/DD/YYYY dates, labels like `CAN $` / `AE DIRHAM` | 37,392 (of 37,877 received) | | CBK mobile payments statistics | HTML table (no export offered) | scrape-only wpDataTable; month names as text; comma-formatted numbers | 231 (monthly, Mar 2007–May 2026) | | GSMA Global Mobile Money Dataset | XLSX | title rows, padding columns, trailing spaces in sheet names, metrics spread across ~100 wide quarter columns | 56,616 (after lossless melt) | **Dropped:** the Kenya Open Data portal (`kenya.opendataforafrica.org`) returned `403 Forbidden` at verification time and is historically unreliable, so it was not used. **Known freshness gap:** CBK's downloadable historical FX file currently ends at 2024-01-03 (current rates live only behind a nonce-protected AJAX table). All published FX values therefore carry explicit `as_of` dates rather than pretending to be current — see the war stories for why we ship the honest gap. ## Live demo The pipeline runs on a weekly schedule against the live sources and publishes its output — if the badge above is green, everything below was built from a real recent run: * **Dashboard** — headline metrics and charts queried from g …