ChachaMarwaDev/kenya-climate-risk-monitor

GCP pipeline for real-time drought and flood risk monitoring across Kenyan counties using open weather APIs, BigQuery, and Looker Studio. # Kenya Climate Risk Monitor ### Drought & Flood Early Warning System ## Problem Statement Kenya's 47 counties face recurring drought and flood crises that affect millions of people, particularly in ASAL (Arid and Semi-Arid) regions. Early warning systems can give communities and authorities days or weeks of advance notice to prepare. This project builds an automated data pipeline that ingests daily weather data for all 47 Kenyan counties, detects anomalies against 40+ year historical baselines, and surfaces risk scores on an interactive dashboard. ## Project Architecture ## Tech Stack ### Cloud Infrastructure ### Data Engineering ### Infrastructure as Code ### Containerization ### Programming Languages ### Key Libraries - `dlt` - Data extraction and loading - `pandas` - Data manipulation - `pyarrow` - Efficient data processing - `google-cloud-bigquery` - BigQuery client - `google-cloud-storage` - Cloud Storage client - `pydantic` - Data validation - `tenacity` - Retry logic - `sqlparse` - SQL parsing ## Project Structure ```text kenya-climate-risk-monitor/ │ ├── .gitignore ├── LICENSE ├── README.md ├── docker-compose.yaml ├── dockerfile │ ├── data/ │ ├── historical/ │ │ └── kenya_weather_2019.csv │ ├── KenyaRegions.json │ └── kenya_counties.csv │ ├── docs/ │ ├── dbt_lineage.jpg │ ├── entity_relationship_diagram.svg │ ├── kenya_weather_gcp_architecture.svg │ ├── kestra_dashboard.jpg │ └── methodology.md │ ├── extraction/ │ ├── .dlt/ │ │ └── config.toml │ ├── .gitignore │ ├── backfill_historical.py │ ├── pipeline_backfill.py │ ├── pipeline_daily.py │ ├── pipeline_dims.py │ ├── pipeline_weather_source.py │ ├── retry_failed.py │ ├── wasp_mining.py │ └── wasp_test.py │ ├── dbt/ │ ├── .gitignore │ ├── dbt_env/ # Python virtual environment (ignored) │ ├── kenya_climate_risk_monitor/ │ │ ├── .gitignore │ │ ├── README.md │ │ ├── dbt_project.yml │ │ ├── analyses/ │ │ …