lawrykoomson/ghana-weather-etl-pipeline

End-to-end weather data pipeline for 5 Ghanaian cities — Airflow orchestrates extraction from Open-Meteo, pandas transforms the data, MySQL stores it, and Power BI visualizes it. # Ghana Weather ETL Pipeline An end-to-end data engineering pipeline that fetches daily weather forecasts for five Ghanaian cities, cleans and transforms the data, loads it into a relational database, and visualizes it in a BI dashboard — orchestrated entirely with Apache Airflow running in Docker. ## Architecture ``` Open-Meteo API --> Airflow DAG --> pandas transform --> MySQL --> Power BI (Extract) (Orchestrate) (Transform) (Load) (Report) ``` **Cities covered:** Accra, Kumasi, Tamale, Takoradi, Cape Coast ## Tech Stack | Layer | Tool | |---|---| | Orchestration | Apache Airflow 3.2.2 (Docker Compose, CeleryExecutor) | | Extract | Open-Meteo API (free, no auth) | | Transform | pandas | | Load | MySQL 8.0 | | Visualization | Power BI Desktop (via ODBC) | | Containerization | Docker Desktop | ## Project Structure ``` agri-weather-pipeline/ ├── dags/ │ └── agriculture_weather_pipeline.py # the DAG: fetch -> transform -> load ├── docker-compose.yaml # Airflow + Postgres + Redis + MySQL services ├── .env # AIRFLOW_UID and pip requirements ├── logs/ # Airflow task logs (gitignored) ├── plugins/ # Airflow plugins (empty, gitignored) └── config/ # Airflow config (gitignored) ``` ## Pipeline Details The DAG (`agriculture_weather_pipeline`) runs daily and consists of three tasks: 1. **`fetch_weather`** — calls the Open-Meteo API for each city's coordinates, pulling a 7-day forecast (max/min temperature, precipitation) 2. **`transform_weather`** — loads the raw JSON into a pandas DataFrame, enforces correct data types, and derives two new columns: `avg_temp` and a boolean `rain_expected` 3. **`load_to_mysql`** — writes the cleaned DataFrame into a `weather_forecasts_ghana` table in MySQL, replacing the table on each run ## Setup & Run 1. Install Docker Desktop 2. Clone this repo and …