aadyagarwal/tea-garden-weather-pipeline

An end-to-end weather data pipeline built with Apache Airflow and PySpark for tea garden operations in Assam, India and Kericho, Kenya. Ingests real hourly weather data, runs PySpark transformations including 7-day rolling averages and window functions, and outputs agricultural flags (frost risk, drought risk, optimal growing conditions) to Parquet # Tea Garden Weather Pipeline 🌿 An end-to-end data pipeline built with **Apache Airflow** and **PySpark** to monitor weather conditions for tea garden operations in **Assam, India** and **Kericho, Kenya**. ## Motivation My family runs a tea garden in Assam. Tea cultivation is highly sensitive to weather — frost destroys crops, drought stunts growth, and optimal temperature/humidity windows are narrow. This pipeline ingests real hourly weather data and outputs actionable agricultural flags to support data-driven farming decisions. ## Architecture ``` Open-Meteo API → Airflow DAG → PySpark Transformations → Parquet Output ``` ## Pipeline Steps 1. **Ingest** — Airflow DAG pulls hourly weather data (temperature, humidity, precipitation) for 3 locations via Open-Meteo API 2. **Transform** — PySpark processes raw data: - Daily aggregations (avg/min/max temperature, total precipitation) - 7-day rolling averages using window functions - Data quality checks (null detection, anomaly flagging) 3. **Output** — Results written to Parquet with agricultural flags: - 🌡️ `frost_risk` — min temp 60% - 🌵 `drought_risk` — 7-day avg precipitation < 2mm ## Locations | Location | Region | |---|---| | Jorhat, Assam | Northeast India tea belt | | Dibrugarh, Assam | Northeast India tea belt | | Kericho, Kenya | East Africa tea growing region | ## Tech Stack - **Apache Airflow** — orchestration, scheduling, retry logic - **PySpark** — distributed data processing, window functions - **Open-Meteo API** — free real-time weather data - **Parquet** — columnar output format for analytics ## Production Considerations - Airflow metadata DB would migrate from SQLite → PostgreSQL for scale - PySpark would run on AWS EMR or Databricks cluster for large datasets - Output Parquet files would land in S3 as part of a data lake architecture - Logs would ship to a centralized observability platform ## Setup ```bash conda create -n weather_pipeline python=3.11 conda activate weather_pipeline pip inst …