declerke/Extract-Africa-Data

# 🌍 ExtractAfricaData: Multi-Sector Climate & Development Pipeline for Africa **ExtractAfricaData** is a production-grade data ingestion pipeline designed to bridge the gap between fragmented public data sources and a clean, queryable PostgreSQL database covering climate, energy, water, biodiversity, and development indicators across all 54 African countries. --- ## 🎯 Project Goal Africa's climate and development data is scattered across dozens of APIs, each with different country names, column formats, and coverage gaps. This pipeline solves that problem by extracting data from the World Bank World Development Indicators (WDI) across seven thematic sectors, standardizing every record into a canonical schema, validating data quality, and loading clean records into PostgreSQL — with a generated validation report per sector documenting what was accepted, what was dropped, and why. The result is a unified, analysis-ready dataset spanning 54 African countries, 63 indicators, and 24 years (2000–2023). --- ## 🧬 System Architecture 1. **Extraction** — `extractors/` — Seven sector-specific modules fetch data from the World Bank WDI API via `wbgapi`. Each extractor targets 9 indicators, 53+ African economies, and the 2000–2023 time range in a single API call. 2. **Transformation** — `clean_transform.py` — Raw wide-format DataFrames are melted to long format, grouped by (country, indicator), and collapsed into standardized record documents with `data_points` keyed by year. 3. **Validation** — `validate.py` — Each record is checked for required fields, valid African ISO3 codes, and non-empty data. Invalid records are logged to a sector validation report. Valid records are upserted into PostgreSQL with `ON CONFLICT DO UPDATE` for idempotency. 4. **Orchestration** — `main.py` — Click CLI supports single-sector runs (`run-sector`) or parallel multi-sector execution (`run-all --workers N`). --- ## 🛠️ Technical Stack | Layer | Tool | Version | |---|---|---| | **Language* …