ianvwyk11/loadshedding-etl

End-to-end ETL pipeline for South African load shedding data using Python, SQL and Apache Airflow # 🔌 South African Load Shedding ETL Pipeline An end-to-end data engineering pipeline that **extracts**, **transforms**, and **loads** South African load shedding schedule data from the EskomSePush API into a structured analytical database — orchestrated with Apache Airflow and scheduled to run daily. --- ## 📐 Architecture ``` EskomSePush API │ ▼ [Extract] ──▶ data/raw/ (raw JSON) │ ▼ [Transform] ──▶ data/processed/ (clean CSVs) │ ▼ [Load] ──▶ SQLite DB (star schema) │ ▼ [Airflow DAG] ── scheduled daily ``` --- ## 🗂️ Project Structure ``` loadshedding_etl/ ├── etl/ │ ├── extract.py # API ingestion │ ├── transform.py # Data cleaning & normalization │ └── load.py # Database loading ├── dags/ │ └── loadshedding_dag.py # Airflow DAG definition ├── sql/ │ ├── create_tables.sql # Schema definition │ └── analysis_queries.sql # Analytical SQL queries ├── data/ │ ├── raw/ # Raw API responses (gitignored) │ └── processed/ # Cleaned CSVs (gitignored) ├── notebooks/ │ └── exploration.ipynb # EDA & findings ├── tests/ │ └── test_transform.py # Unit tests ├── .github/workflows/ │ └── ci.yml # GitHub Actions CI ├── requirements.txt ├── .env.example └── README.md ``` --- ## ⚙️ Setup ### 1. Clone the repository ```bash git clone github.com cd loadshedding-etl ``` ### 2. Create virtual environment ```bash python -m venv venv source venv/bin/activate # Linux/macOS venv\Scripts\activate # Windows ``` ### 3. Install dependencies ```bash pip install -r requirements.txt ``` ### 4. Configure environment variables ```bash cp .env.example .env # Add your EskomSePush API key to .env ``` ### 5. Initialise the database ```bash python -m etl.load --init ``` ### 6. Run the pipeline manually ```bash python -m etl.extract python -m etl.transform python -m etl.load ``` --- ## 🗄️ Database Schema The pipeline loads data i …