Jayson-gor/kenya-weather-data-pipeline

A complete weather data engineering project that automates the collection, storage, and visualization of real-time and historical weather data across 10 major towns in Kenya. # kenya-weather-data-pipeline A complete weather data engineering project that automates the collection, storage, and visualization of real-time and historical weather data across 10 major towns in Kenya. This project uses Apache Airflow to fetch, process, and store weather data for Kenyan towns, with visualization in Metabase. ## 🧰 Tech Stack | Component | Tool/Tech | |---------------|-----------------------------| | Workflow Orchestration | Apache Airflow (Dockerized) | | Data Collection | Python, OpenWeather & Weatherbit APIs | | Database | PostgreSQL | | Visualization | Metabase (Dockerized) | ## 🌐 Tracked Towns Nairobi, Mombasa, Kisumu, Nakuru, Eldoret, Nyeri, Kakamega, Thika, Garissa, Kitale --- ## 📊 Features ✅ Hourly streaming of current weather data ✅ Batch loading of previous day's weather (morning & evening) ✅ PostgreSQL storage with duplicate protection ✅ Scheduled using Airflow ✅ Interactive dashboard built with Metabase ✅ Insights like temperature trends, humidity scatter plots, heatmaps, and more --- ## 🏗️ Architecture +-----------------+ +-----------------+ | Airflow DAGs | -----> | Weather APIs | +-----------------+ +-----------------+ | v +-------------------+ | Python Scripts | +-------------------+ | v +-------------------+ | PostgreSQL DB | +-------------------+ | v +-------------------+ | Metabase | +-------------------+ ⚙️ Setup Instructions 🐳 1. Clone and Start Docker Compose _git clone github.com /kenya-weather-pipeline.git cd kenya-weather-pipeline docker compose up -d 🗃️ 2. Connect to PostgreSQL _docker exec -it airflow-postgres-1 psql -U postgres CREATE DATABASE kenya_weather; Run the table creation SQL provided above inside this DB. 🌀 Airflow DAG: kenya_weather_pipeline Located in: dags/kenya_weather_dag.py Tasks: batch_load_historical_data: Pulls weather data from Weatherbit for yeste …