Daisy254-can/kenya-food-prices-etl
Domaine:
agricultureType de record:
software
Créateur:
Dai
Hôte:
Production-ready ETL pipeline for WFP Kenya food prices data
# Kenya Food Prices — Data Engineering Capstone Project
**March 2026 | Month 2 Deliverable**
*"Cleaned & Enriched Food Prices with Visualization Prep & Production-Ready ETL Pipeline"*
---
## Table of Contents
1. Project Overview
2. Dataset
3. Pipeline Architecture
4. Project Structure
5. Setup & Installation
6. Running the Pipeline
7. Star Schema Design
8. dbt Models
9. Airflow DAG
10. Visualizations
11. Data Issues & Design Choices
12. Challenges & Learnings
13. Future Ideas
---
## Project Overview
This project builds a production-ready ETL pipeline that ingests, cleans, models,
and visualises monthly food price data for Kenya from the World Food Programme (WFP).
**Business question:** How have food prices across Kenya's markets changed over time,
and which commodities and regions experience the greatest price volatility?
**Key deliverables:**
- Modular Python ETL pipeline (extract → clean → quality check → load)
- PostgreSQL star schema (dim_date, dim_market, dim_commodity, fact_prices)
- dbt Core models for staging and analytical marts
- Airflow DAG for monthly orchestration
- Grafana/Metabase-ready analytical SQL queries
- Interactive data visualisations
---
## Dataset
| Property | Value |
|----------|-------|
| Source | WFP Humanitarian Data Exchange (HDX) |
| File | `wfp_food_prices_ken.csv` |
| Rows | ~17,994 |
| Columns | 16 |
| Date range | January 2006 – September 2025 |
| Granularity | Monthly price per commodity per market |
| Currency | KES (Kenyan Shilling) + USD equivalent |
**Key columns:**
| Column | Description |
|--------|-------------|
| `date` | Price observation date (monthly) |
| `admin1` / `admin2` | Province and district |
| `market` / `market_id` | Market name and WFP ID |
| `commodity` / `commodity_id` | Food item name and WFP ID |
| `unit` | Unit of measure (KG, 90 KG, 50 KG, L, etc.) |
| `pricetype` | Retail or Wholesale |
| `price` | Price in KES |
| `usdprice` | Price in USD |
---
## Pipeline Architecture
```
┌───────── …