Jenn-mawia/Kenya-Food-Prices-Data-Engineering-Project

Data Engineering project about WFP Kenya Food Prices. ETL process, Airflow DAG # Kenya Food Prices — Data Engineering Capstone Project **Dataset:** WFP (World Food Programme) Kenya Food Prices **Coverage:** January 2006 – March 2026 | 18,837 rows | 16 columns **Tools:** Python · PostgreSQL · Apache Airflow · dbt · Metabase · Docker --- ## Project Overview This project builds a production-style data engineering pipeline for the WFP Kenya Food Prices dataset. It is structured in two parts: - **Part 1 (February 2026):** Raw data staging — PostgreSQL setup, CSV loading, SQL analysis queries, and a basic Airflow DAG - **Part 2 (March 2026):** Cleaned ETL pipeline — modular Python scripts, star schema, dbt models, Metabase dashboards, and an updated Airflow DAG --- ## Repository Structure ``` DE_PROJECT/ │ ├── 01_create_database.sql # Part 1: Create food_prices database ├── 02_setup.sql # Part 1: Create staging.raw_food_prices table ├── 03_load_csv.py # Part 1: Load CSV into PostgreSQL ├── 04_analysis_queries.sql # Part 1: 7 analytical SQL queries ├── clean.py # Part 1: Early cleaning experiments ├── dags/ │ └── airflow_dag.py # Part 1: Airflow DAG (3 tasks) ├── docker-compose.yaml # Docker setup for Airflow (Part 1) ├── .env # Airflow UID │ └── DE_PROJECT_part2/ # Part 2 — full ETL pipeline ├── etl/ │ ├── __init__.py │ ├── extract.py # Extract from URL or local CSV │ ├── clean.py # Full pandas cleaning pipeline │ └── load.py # Load to PostgreSQL ├── sql/ │ └── 02_cleaned_schema.sql # Cleaned table + dimension tables ├── dbt_project/ │ ├── dbt_project.yml │ ├── profiles.yml │ ├── macros/ │ │ └── generate_schema_name.sql │ ├── models/ │ │ ├── schema.yml │ │ ├── staging/ │ │ │ ├── sources.yml │ │ │ ├── stg_food_prices.sql │ │ │ ├── dim_date.sql │ │ │ ├── dim_market.sql │ │ │ ├── dim_commodity.sql │ │ …