BrianOyollo/Energy-Data-Extraction-MongoDB-Storage

Energy data (2000–2024) extraction & MongoDB storage from Africa Energy Portal # Energy Data Extraction & MongoDB Storage (2000–2024) ## Overview This project automates the **extraction, formatting, and storage** of energy-related data from the Africa Energy Portal (AEP) It retrieves key energy indicators for **all African countries** from **2000 to 2024** and stores them in a **MongoDB collection**. The task was completed as part of **Week 1 of the Internship Program**, focusing on **data engineering fundamentals** — web data extraction, transformation, and database integration. ## Objectives 1. **Data Extraction** – Collect energy data from AEP dashboards, datasets, and country profiles. 2. **Data Transformation** – Structure all data into a standardized schema. 3. **Data Storage** – Store the formatted data in MongoDB using Dockerized containers. 4. **Validation** – Ensure consistency, completeness, and proper data typing across all records. ### Extracted Datasets Include: - Electricity generation - Electricity access - Energy consumption - Renewable energy usage - Clean cooking access - Energy efficiency indicators ### Data Coverage: - **Countries:** All African nations - **Years:** 2000 → 2024 - **Sources:** - Country Profile | Africa Energy Portal - AEP Database | Africa Energy Portal ## Data Format Each document represents **one metric for one country** across all years. | Field | Description | | --- | --- | | `country` | Country name | | `country_serial` | Country code or serial number | | `metric` | Energy indicator name | | `unit` | Measurement unit | | `sector` | Main category (e.g., Electricity, Energy Access) | | `sub_sector` | Subcategory within sector | | `sub_sub_sector` | Further subcategory if applicable | | `source_link` | URL of the data source | | `source` | Data source name | | `2000 ... 2024` | Yearly values for the metric | ### Example schema ```json { "country": "Kenya", "country_serial": 404, "metric": "Electricity access (% of population)", "unit": "% of population", "sector": "Electricity …