jude-craft/kenya-agri-climate-data-pipeline

End-to-end data engineering pipeline integrating 20 years of Kenyan climate and agricultural market data using Python, Kestra, GCS, BigQuery, and dbt to analyze how extreme weather impacts commodity prices and food supply chains. # Kenya Agri-Market & Climate Impact Pipeline > An end-to-end batch data engineering pipeline I built to analyze the relationship between climate variability and agricultural commodity prices across Kenya. --- ## Table of Contents - Problem Statement - Objective - Architecture Overview - Tech Stack - Project Structure - Pipeline Flow (Deep Dive) - Phase 1 — Extract - Phase 2 — Load to Data Lake (GCS) - Phase 3 — Load to Data Warehouse (BigQuery Bronze) - Phase 4 — Transform (dbt) - Data Sources & Schema - Weather Data (Producer Hubs) - Market Data (Consumer Hubs) - Infrastructure as Code (Terraform) - Orchestration (Kestra) - Architecture Decision Records (ADR) - Getting Started - Prerequisites - 1. Clone the Repository - 2. GCP Credentials Setup - 3. Provision Infrastructure with Terraform - 4. Start the Orchestration Stack - 5. Deploy the Pipeline Flow to Kestra - Lessons Learned & Engineering Notes - Future Work --- ## Problem Statement Agriculture is the backbone of Kenya's economy — it employs over 70% of the rural population and directly influences the country's GDP(Gross Domestic Product), food security, and regional stability. Yet, it is one of the sectors most vulnerable to extreme climate events. A single prolonged drought in the Rift Valley can cascade into supply shortages in Nairobi markets within weeks, driving food inflation that hits the lowest-income households the hardest. Despite this, the relationship between **climate patterns** (rainfall, temperature, evapotranspiration) and **commodity price fluctuations** remains poorly quantified at scale. The data exists — scattered across weather APIs and humanitarian datasets — but no unified, analytics-ready pipeline connects these two domains for Kenya. ## Objective I designed this pipeline to solve that gap. Specifically, it: 1. **Extracts** 20+ years of daily historical weather data for Kenya's key **producer counties** (the agricultural heartland where food is grown). 2. **Extracts** month …