data-with-joe/electricity_generation

comparing electricity generation in Europe and Africa using data from Ember-Energy # electricity_generation ## Overview This project is an end-to-end data engineering pipeline that ingests global electricity generation data from the Ember Energy API, processes it using PySpark, and stores it in a Delta Lake table for analytics and dashboarding. The goal is to analyze and compare electricity generation across Africa and Europe, broken down by energy source (Solar, Wind, Coal, Bioenergy, etc.) and measured in TWh (terawatt-hours). ## Objectives Ingest electricity generation data from an external API Handle multi-country and paginated API responses Transform and clean raw data using PySpark Enrich dataset with region mapping (Africa vs Europe) Store processed data in Delta Lake (Unity Catalog) Enable analytics-ready datasets for dashboards ## Architecture Ember Energy API Python (requests) Raw JSON response PySpark Transformation Layer Data Cleaning + Region Mapping Delta Lake (Bronze Table) Analytics / Dashboard (Power BI / Databricks SQL) ## Tech Stack Python (requests, pandas) PySpark (Databricks) Delta Lake Databricks Unity Catalog Ember Energy API SQL (for exploration & dashboards) ## Dataset Description Each record represents electricity generation for a country by energy source. Key Columns: Column Description entity Country name entity_code ISO country code date Year of observation source Energy type (Solar, Wind, Coal, etc.) generation_twh Electricity generated (TWh) share_of_generation_pct Percentage share of total generation region Africa or Europe ingestion_time Timestamp of data load