declerke/Kenya-Economic-Pulse

# 🇰🇪 Kenya Economic Pulse: Delta Lakehouse Pipeline **Kenya Economic Pulse** is a production-grade Delta Lakehouse pipeline that ingests 8 macroeconomic indicators for Kenya (1960–2025) from the World Bank Open Data API, processes them through a Bronze → Silver → Gold medallion architecture on MinIO using PySpark and Delta Lake, transforms the data into 6 analytical dbt models backed by 28 tests, and serves 4 interactive Evidence.dev dashboard pages — all orchestrated by Apache Airflow and containerised with Docker Compose. --- ## 🎯 Project Goal Kenya's macroeconomic story — GDP growth, inflation, unemployment, trade, population — is publicly available through the World Bank API but scattered across individual indicator endpoints with no unified analytical view. Raw JSON responses contain mixed types, sparse historical values, and no temporal trend analysis. Kenya Economic Pulse solves this by running a weekly Airflow pipeline that pulls all 8 indicators in a single DAG run, writes immutable raw JSON to a Bronze layer on MinIO, transforms it into a typed and deduplicated Delta Lake Silver table using PySpark, runs 6 dbt Gold models with rolling averages and derived metrics, and renders the results as a multi-page Evidence.dev dashboard — giving analysts a single URL to explore 66 years of Kenya's economic history. --- ## 🧬 System Architecture 1. **Ingestion (World Bank API)** — `fetch_indicators` pulls 8 indicator codes for Kenya via the REST API (`/v2/country/KE/indicator/{code}?per_page=1000`), normalises the response envelope, and pushes raw records to Airflow XCom 2. **Bronze Layer (MinIO)** — `save_to_bronze` writes one JSON file per indicator to `s3://bronze/year=YYYY/month=MM/day=DD/` — immutable, never overwritten, partitioned by run date 3. **Silver Layer (PySpark + Delta Lake)** — `spark_bronze_to_silver` reads all Bronze JSON, casts types, flags null values, deduplicates on `(indicator_code, year)`, and writes a full-overwrite Delta Lake table to …