vikToure/Daily_sales_WFP_ETL_pipeline_project

Python ETL pipeline on 87K+ WFP food price records — tracks Nigerian commodity trends, regional price volatility, and how food prices in Borno/Yobe diverged from the rest of Nigeria after the 2009 insurgency. # WFP Food Prices in Nigeria **3MTT NextGen Cohort — Data Science Track (DS-14)** **Author:** Victor Egenkonye ## Project Overview This project builds a complete ETL (Extract, Transform, Load) pipeline analyzing food commodity prices across Nigeria over more than two decades, using real World Food Programme (WFP) market monitoring data. **Business questions:** 1. How have prices for key food commodities changed over time? 2. Which Nigerian states have the highest and most volatile food prices? 3. Did food prices in the insurgency-affected northeast (Borno/Yobe) diverge from the rest of the country after the Boko Haram insurgency escalated in 2009? ## Data Source - **Dataset:** WFP Food Prices for Nigeria - **Source:** World Food Programme (WFP), via the Humanitarian Data Exchange (HDX) — `data.humdata.org` - **Size:** 87,963 raw price records, 16 columns, spanning January 2002 – April 2026 - **Coverage:** Multiple states, markets, and food commodities across Nigeria, with both Retail and Wholesale price types ## Tools & Libraries | Tool | Purpose | |---|---| | Python 3 | Core language | | pandas | Data cleaning, transformation, aggregation | | numpy | Numerical operations, conditional column creation | | matplotlib | Visualization | | sqlite3 | Loading cleaned data into a persistent database | | Jupyter Notebook | Development environment | ## Key Data Quality Decisions This dataset required more careful handling than a typical clean CSV — several deliberate scoping decisions were made along the way, each documented here for transparency: - **Price verification filter:** Only rows flagged `'actual'` in `priceflag` were kept (51,789 of 87,963 rows). Rows flagged `'aggregate'` or `'actual,aggregate'` were excluded, since these represent estimated/summarized values rather than directly observed prices. This disproportionately affects **Borno (14,574 excluded rows) and Yobe (9,586 excluded rows)** — a reflection of genuine d …