Jawadx7/Ghana-Food-Market-Price-Analytics-Pipeline
Domaine:
agricultureType de record:
software
Créateur:
Jaw
Hôte:
A data engineering pipeline that ingests, cleans, transforms, and validates Ghana’s food market price data using Python, pandas, Airflow, PyTest, and automated logging. Includes modular ETL architecture, unit tests, error handling, and orchestration.
# Ghana Food Market Price Analytics Pipeline ✅
Lightweight ETL pipeline to extract, transform and (eventually) load WFP market price data for Ghana. The project focuses on cleaning and standardizing food price data (per-kg prices), detecting outliers, and preparing the dataset for downstream analysis and analytics.
---
## Table of Contents
- 📌 About
- ⚙️ Features
- 🗂️ Repository Structure
- 📥 Data
- ⚙️ Installation
- 🚀 Usage
- 🛠️ Development Notes
- ✅ Tests
- ✍️ Contributing
- 📄 License
---
## 📌 About
This repo implements an ETL pipeline for WFP Ghana food price data (CSV). The pipeline reads raw CSV(s), runs a set of data cleaning and transformation steps, detects outliers and provides an enriched dataset ready for analytics.
## ⚙️ Features
- Extraction from CSV using `pandas` (handles HXL style header row)
- Robust transformation steps:
- Remove HXL header row
- Standardize column names
- Parse dates reliably
- Normalize categories and text fields
- Convert numeric and coordinate types
- Normalize units to per-kg and compute `price_per_kg`
- Detect outliers using z-score method
- Structured logging (writes component logs to `logs/`)
- Modular code organized under `src/`
## 🗂️ Repository Structure
Key files and folders:
- `src/pipeline.py` — main entry point that orchestrates extraction, transform, enrichment
- `src/extract/csv_extractor.py` — reads CSV into a DataFrame
- `src/transform/clean_prices.py` — transformation helpers (remove HXL, clean types, normalize units, outlier detection)
- `src/transform/transform_prices.py` — orchestrator for transformation steps
- `src/transform/enrich_prices.py` — enrichment placeholder (add more features here)
- `src/config/logger_config.py` — sets up file and console logging
- `data/raw/wfp_food_prices_gha.csv` — example raw dataset (included)
- `requirements.txt` — Python dependencies
---
## 📥 Data
Source data should be a CSV similar to `data/raw/wfp_food_prices_gha.csv`. The file included in `data/raw/` conta …