Lucy23-2024/africa_energy_project
Domain:
environment and energyRecord type:
dataset
Creator:
Luc
Host:
Energy data extraction and MongoDB storage for African countries (2000–2024)
# Energy Data Extraction and MongoDB Storage (2000–2024)
## Overview
This project extracts **energy-related data for African countries (2000–2024)** from the **Africa Energy Portal (AEP)**, formats it according to a standard schema, and stores it in **MongoDB**.
It was completed as part of the **Internship Week 1** task on *Energy Data Extraction and MongoDB Storage*.
---
## Project Structure
```
AFRICA_ENERGY_PROJECT/
│
├── venv/ # Virtual environment folder
├── .gitignore # Files and folders to ignore in GitHub
├── .python-version # Python version configuration
├── africa_energy_data.ipynb # Main Jupyter Notebook (data processing workflow)
├── africa_energy_data.csv # Final processed dataset (ready for MongoDB)
├── electricity.json # Raw JSON data (electricity indicators)
├── energy.json # Raw JSON data (energy indicators)
├── social_and_economic.json # Raw JSON data (social and economic indicators)
├── main.py # Optional script version of notebook logic
├── pyproject.toml # Python project dependencies/config
└── README.md # Project documentation (this file)
```
*Note:* The `.json` files are the manually downloaded datasets from the Africa Energy Portal.
They are combined and cleaned within the Jupyter notebook.
---
## Project Workflow
### 1. **Data Collection**
- Data was obtained manually from the **Africa Energy Portal**.
- Using browser developer tools, the **Network → Response** tab revealed API responses in JSON format.
- The JSON data was **copied**, **saved in Notepad**, and stored as `.json` files.
- These files were then processed using Python.
---
### 2. **Data Processing Steps**
| Step | Description |
|------|--------------|
| **1. Import Libraries & Load Data** | Used `glob` to load all JSON files and combine them into one dataset. |
| **2. Rename Columns** | Standardized field nam …