RonohSoy/Africa_Energy_Data_Pipeline

This project automates the extraction, transformation, and storage of energy-related datasets for all African countries (2000–2024). # Africa Energy Data Pipeline The **Africa Energy Data Pipeline** is an end-to-end ETL system that automates the collection, transformation, validation, and storage of African energy sector data from the Africa Energy Portal. It standardizes datasets across all **54 African countries** for the years **2000–2022**, covering the subsectors: **Access**, **Supply**, and **Technical**. ## Overview This project was developed as part of a data engineering internship task focused on building a **data engineering workflow** for large-scale energy datasets. The pipeline integrates cloud data extraction, validation, and cloud database storage using **MongoDB Atlas**, making the data ready for **visualization in Power BI or Tableau**. ## Key Features - Automated extraction of energy datasets from the Africa Energy Portal API - Data transformation and normalization across countries and subsectors - Validation checks for missing years, duplicates, and data consistency - Cloud storage using MongoDB Atlas - Output structured for direct visualization in BI tools ## Tech Stack | Category | Tools | |-----------|-------| | **Language** | Python 3.12 | | **Database** | MongoDB Atlas | | **Libraries** | `requests`, `pandas`, `pymongo`, `json` | | **IDE** | PyCharm | | **Visualization (Optional)** | Power BI, Tableau | ## Project Structure ``` Africa_Energy_Data_Pipeline/ │ ├── AFDP.py # Main ETL pipeline script ├── africa_energy_data.json # Raw extracted data ├── formatted_africa_energy_data.json # Transformed and formatted data ├── validation_report.json # Data validation report ├── .gitignore # Ignored files └── README.md # Project documentation ``` ## Setup Instructions ### How to Run ### 1️. Clone the Repository git clone github.com cd Africa_Energy_Data_Pipeline ### 2. Create a Virtual Environment #### Create a virtual …