KellyKiprop/Africa-energy-portal-pipeline

# **Africa Energy Portal Data Pipeline** A comprehensive ELT (Extract, Load, Transform) pipeline for extracting, processing, and analyzing energy data from the Africa Energy Portal. This project collects energy indicators for all 54 African countries from 2000-2024 and stores them in multiple formats including MongoDB. ## 📊 Project Overview This project automates the extraction of energy-related data from the Africa Energy Portal, covering all 54 African countries with comprehensive energy metrics from 2000 to 2024. The data is processed, validated, and stored in both file-based formats (CSV, JSON) and database systems (MongoDB). ### Key Features - **54 African Countries** complete coverage - **33 Energy Metrics** per country including electricity access, generation, consumption, renewables - **25-Year Time Series** from 2000 to 2024 - **Multiple Output Formats** CSV, JSON, MongoDB - **Data Validation** with comprehensive quality reports - **Flexible Deployment** Local or Cloud MongoDB options ## 🏗️ Project Structure ``` aep_pipeline/ ├── extract_energy_data.py # Main data extraction script ├── load.py # MongoDB Cloud loading script ├── requirements.txt # Python dependencies ├── africa_energy_data_2000_2024.csv # Output dataset (CSV) ├── africa_energy_data_2000_2024.json # Output dataset (JSON) └── validation_report.json # Data quality report ``` ## 📈 Data Schema Each record contains the following structure: | Column | Type | Description | |--------|------|-------------| | `country` | String | Country name | | `country_serial` | Integer | Country identifier (1-54) | | `metric` | String | Energy metric name | | `unit` | String | Measurement unit (%, MW, GWh, etc.) | | `sector` | String | Energy sector category | | `sub_sector` | String | Sub-sector classification | | `sub_sub_sector` | String | Detailed category | | `source_link` | String | Data source URL | | `source` | String | Data source name | | `2000` …