MBabu215/African-Energy-data-scraping

# ⚡ African Energy Data Scraper (2000–Present) This project automates the **scraping, preprocessing, and storage** of **energy-related data** from the **African Energy Portal (AEP)**. It retrieves structured indicators — such as electricity generation, consumption, renewables, and access rates — covering **2000 to the present**, then stores them in a **MongoDB database** for analysis and visualization. --- ## 📚 Table of Contents 1. Overview 2. Features 3. Project Structure 4. Setup Instructions 5. Environment Variables 6. Workflow Summary 7. MongoDB Integration 8. Chrome/Brave Launcher Configuration 9. Sample Outputs 10. Troubleshooting 11. Future Improvements 12. License --- ## 📖 Overview The **African Energy Data Scraper** automatically extracts energy-related statistics from the **African Energy Portal** (AEP) and organizes them into a clean, analysis-ready format. It includes scripts for: - Scraping and collecting raw data (JSON format) - Preprocessing and cleaning data into CSV (wide format) - Uploading the final dataset into MongoDB The result is a structured, time-series dataset ideal for **data analysis, dashboards, and machine learning models**. --- ## ✨ Features ✅ Automated scraping from the African Energy Portal ✅ Time range: **2000–present** ✅ Saves raw JSON responses for reproducibility ✅ Preprocessing into wide-format CSV for modeling ✅ Automatic upload to MongoDB collection ✅ Supports switching between **Chrome** and **Brave** browser launchers ✅ Secure `.env` configuration for MongoDB credentials --- ## 🗂 Project Structure ``` African_Energy_Scraping/ │ ├── requirements.txt # Python dependencies ├── .env # Environment variables (excluded in .gitignore) ├── README.md # Documentation (this file) │ ├── notebooks/ │ ├── energy.ipynb # Notebook for exploration or visualization │ ├── aep_preprocessed_wide_2000_2022.csv # Processed dataset │ └── scraped_json/ # Folder for storing raw scraped JSON files │ └── Scripts/ ├── main.py # Entry point – …