brad-kipkoech/africa-energy-data

project where I scraped data from africa energy data portal and saved in mongodb # Africa Energy Data Pipeline This project automates the collection, cleaning, transformation, and storage of energy-related data across African countries. Data is scraped from the **Africa Energy Portal** using **Selenium**, cleaned and transformed with **Pandas**, and then stored in **MongoDB Atlas** for analysis and visualization. # Project Workflow ## Data Collection - Tool: **Selenium** - Source: Africa Energy Portal - Script: `scripts/scrape_africa_energy.py` - Output: `data/raw_data.csv` ## Data Cleaning & Formatting - Tool: **Pandas** - Notebook: `notebooks/cleaning_notebook.ipynb` - Steps: - Handle missing values - Standardize metric names and units - Reshape data into the required format: ``` ["country", "country_serial", "metric", "unit", "sector", "sub_sector", "sub_sub_sector", "source_link", "source", "2000", "2001", ..., "2024"] ``` - Output: `data/cleaned.csv` ## MongoDB Integration - Script: `scripts/upload_formatted_data.py` - Database: **MongoDB Atlas** - Collection: `energy_db.africa_energy` - Schema example: ```json { "id": "kenya_26_energy_power_generation_general_installed_capacity_mw", "country": "Kenya", "data_points": { "2015": 2300, "2016": 2400, "2024": 3778.5 }, "metric": "Installed Capacity", "sector": "Energy", "unit": "MW", "source_link": "ENERGY & PETROLEUM STATISTICS REPORT", "sources": "Epra 2023-2024 FY Reports" } ``` # Technologies Used Python 3.10+ Selenium Pandas PyMongo MongoDB Atlas dotenv # setup instructions git clone github.com cd africa-energy-data python -m venv venv linux (source venv/bin/activate) (Windows: venv\Scripts\activate) pip install -r requirements.txt