hajirufai/afridata-pipeline

Production-grade ETL pipeline for African economic data. World Bank API → DuckDB star-schema warehouse → Interactive analytics dashboard. Dimensional modeling, automated data quality, and Vercel deployment. # 🌍 AfriData Pipeline **A production-grade data engineering project** that extracts economic indicators for all 54 African countries from the World Bank API, transforms and loads them into a DuckDB analytical warehouse with dimensional modeling, runs automated data quality checks, and serves a beautiful interactive dashboard. --- ## ✨ Features | Feature | Description | |---------|-------------| | 🔄 **ETL Pipeline** | Extracts 13,500+ records from World Bank API v2 with retry logic | | 🏗️ **Star Schema** | Dimensional model in DuckDB (fact + 3 dimensions) | | ✅ **Data Quality** | Automated completeness, validity, and freshness checks | | 📊 **Dashboard** | Interactive charts, choropleth map, country profiles | | 🌙 **Dark Mode** | Full dark/light theme support | | ⚡ **Daily Refresh** | GitHub Actions cron updates data every morning | ## 📊 Dashboard Preview The dashboard features: - **KPI Summary Cards** — Total GDP, population, average growth, life expectancy - **Choropleth Map** — Color-coded Africa map for any indicator (Leaflet.js) - **Country Comparison** — Compare up to 6 countries over 25 years - **Rankings Table** — Sortable rankings across all 10 indicators - **Quality Scorecard** — Transparency on data completeness and freshness - **Country Profiles** — Click any country for a detailed modal view ## 🏗️ Architecture ``` ┌────────────────────────┐ │ World Bank API v2 │ Free, no auth, 16K+ indicators └──────────┬─────────────┘ │ Extract (httpx + retry) ▼ ┌────────────────────────┐ │ Transform (Python) │ Clean, enrich, calculate YoY change └──────────┬─────────────┘ │ Load ▼ ┌────────────────────────┐ │ DuckDB Warehouse │ Star schema: fact_indicators + 3 dims │ (Dimensional Model) │ └──────────┬─────────────┘ │ Export JSON ▼ ┌────────────────────────┐ │ Static Dashboard │ HTML + Tailwind + Chart.js + Leaflet │ (Deployed on Vercel) │ └────────────────────────┘ ``` ## 📈 Data Coverage | Indicator | Category | Unit | |------ …