Jayson-gor/kenya-weather-data-pipeline
Domaine:
climateType de record:
project
Créateur:
Jay
Hôte:
A complete weather data engineering project that automates the collection, storage, and visualization of real-time and historical weather data across 10 major towns in Kenya.
# kenya-weather-data-pipeline
A complete weather data engineering project that automates the collection, storage, and visualization of real-time and historical weather data across 10 major towns in Kenya.
This project uses Apache Airflow to fetch, process, and store weather data for Kenyan towns, with visualization in Metabase.
## 🧰 Tech Stack
| Component | Tool/Tech |
|---------------|-----------------------------|
| Workflow Orchestration | Apache Airflow (Dockerized) |
| Data Collection | Python, OpenWeather & Weatherbit APIs |
| Database | PostgreSQL |
| Visualization | Metabase (Dockerized) |
## 🌐 Tracked Towns
Nairobi, Mombasa, Kisumu, Nakuru, Eldoret, Nyeri, Kakamega, Thika, Garissa, Kitale
---
## 📊 Features
✅ Hourly streaming of current weather data
✅ Batch loading of previous day's weather (morning & evening)
✅ PostgreSQL storage with duplicate protection
✅ Scheduled using Airflow
✅ Interactive dashboard built with Metabase
✅ Insights like temperature trends, humidity scatter plots, heatmaps, and more
---
## 🏗️ Architecture
+-----------------+ +-----------------+
| Airflow DAGs | -----> | Weather APIs |
+-----------------+ +-----------------+
|
v
+-------------------+
| Python Scripts |
+-------------------+
|
v
+-------------------+
| PostgreSQL DB |
+-------------------+
|
v
+-------------------+
| Metabase |
+-------------------+
⚙️ Setup Instructions
🐳 1. Clone and Start Docker Compose
_git clone github.com /kenya-weather-pipeline.git
cd kenya-weather-pipeline
docker compose up -d
🗃️ 2. Connect to PostgreSQL
_docker exec -it airflow-postgres-1 psql -U postgres
CREATE DATABASE kenya_weather;
Run the table creation SQL provided above inside this DB.
🌀 Airflow DAG: kenya_weather_pipeline
Located in: dags/kenya_weather_dag.py
Tasks:
batch_load_historical_data: Pulls weather data from Weatherbit for yeste …