HezekiahEnahoro/African-Development-Data-Warehouse

A production-grade data pipeline that extracts 16 development indicators for 30 African countries from the World Bank and WHO APIs, loads them into Amazon Redshift Serverless, and serves a Grafana dashboard for exploration. # African Development Data Warehouse A production-grade data pipeline that extracts 16 development indicators for 30 African countries from the World Bank and WHO APIs, loads them into Amazon Redshift Serverless, and serves a Grafana dashboard for exploration. --- ## Architecture ```mermaid flowchart LR WB["🌍 World Bank API\n(16 indicators)"] --> EWB["extract_worldbank"] WHO["🏥 WHO GHO API\n(5 indicators)"] --> EWHO["extract_who"] EWB --> S3[("Amazon S3\nParquet / Snappy")] EWHO --> S3 EWB --> LOAD["load_to_redshift"] EWHO --> LOAD S3 --> LOAD LOAD --> RAW[("Redshift\nraw schema")] RAW --> TRANS["run_transforms"] TRANS --> ANA[("Redshift\nanalytics schema")] ANA --> QC["run_data_quality_checks"] ANA --> GF["📊 Grafana\nlocalhost:3000"] ``` *Orchestrated by Apache Airflow 2.8 — runs every Monday at 02:00 UTC* **Data flow per run:** 1. World Bank API → Parquet → `s3://bucket/staging/worldbank/` 2. WHO GHO API → Parquet → `s3://bucket/staging/who/` 3. Redshift COPY → `raw.wb_countries`, `raw.wb_indicators`, `raw.who_indicators` 4. SQL transforms → `analytics.dim_country`, `analytics.dim_indicator`, `analytics.fct_indicators`, `analytics.mart_health_summary`, `analytics.mart_economic_summary` --- ## Screenshots *Health and economic indicators across 30 African countries — Grafana 10.4* *Weekly pipeline — parallel extraction, incremental load, transforms, quality checks* --- ## Incremental Loading `wb_indicators` uses an incremental pattern to avoid re-fetching 20+ years of data on every run: 1. Query `MAX(year)` from `raw.wb_indicators` in Redshift 2. Re-fetch the last **2 years** (`INCREMENTAL_LOOKBACK = 2`) to catch late-arriving World Bank updates 3. `DELETE FROM raw.wb_indicators WHERE year >= since_year` 4. COPY the fresh Parquet files from S3 `wb_countries` (30 rows, static metadata) and `who_indicators` (WHO API has no date filter) are always full-reloaded. The `since_year` value is passed between Airflow tasks via XCom so the load step uses …