noelle-mburu/africa-energy-portal

Automated data pipeline for African energy statistics (2000-2022) # Africa Energy Portal - Data Pipeline Automated data extraction and processing pipeline for African energy statistics (2000-2024). ## Project Overview This project builds a complete data pipeline that: - Scrapes energy data from the Africa Energy Portal - Parses and processes 69+ Excel files - Transforms data into a structured format - Stores 50,000+ data points in MongoDB Atlas - Validates data quality (97-99% completeness) ## Data Coverage - **Countries:** 54 African nations - **Time Period:** 2000-2024 - **Categories:** - Electricity - Energy - Social and Economic - **Data Points:** 50,000+ - **Completeness:** 97-99% across all sectors ## Tech Stack - **Python 3.x** - **Selenium** - Web scraping & automation - **Pandas** - Data parsing & transformation - **MongoDB Atlas** - Cloud database storage - **Python-dotenv** - Environment variable management ## Project Structure ``` ├── scraper.py # Web scraping module ├── parser.py # Excel file parser ├── transformer.py # Data transformation ├── mongodb_uploader.py # Database operations ├── validator.py # Data validation ├── config.py # Configuration settings ├── requirements.txt # Python dependencies └── validation_report.txt # Data quality report ``` ## Installation ### Prerequisites - Python 3.8+ - Chrome browser - MongoDB Atlas account ### Setup 1. **Clone the repository** ```bash git clone github.com cd africa-energy-portal ``` 2. **Create virtual environment** ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate ``` 3. **Install dependencies** ```bash pip install -r requirements.txt ``` 4. **Configure environment variables** Create a `.env` file: ``` MONGODB_URI=your_mongodb_connection_string DATABASE_NAME=your_database_name ``` ## Usage ### Run Complete Pipeline ```bash # 1. Scrape data from Africa Energy Portal python scraper.py # 2. Parse downloaded Excel files …