lawrykoomson/Ghana-Momo-ETL-Pipeline

# 🇬🇭 Ghana Mobile Money ETL Pipeline A production-grade **Extract → Transform → Load (ETL)** pipeline that processes Ghana Mobile Money (MoMo) transaction data, cleans it, flags anomalies, and loads results into a live **PostgreSQL** data warehouse — with a full **dbt analytical layer**, **Power BI dashboard**, **Airflow DAG**, and **Kafka stream simulator**. Built to mirror real-world data engineering workflows used at fintech companies like **Hubtel** and telecoms like **MTN Ghana**. --- ## 🏗️ System Architecture ``` [Raw CSV / Synthetic Data Source] │ ▼ ┌───────────┐ │ EXTRACT │ ← Reads CSV or generates 10,000 synthetic MoMo transactions └───────────┘ │ ▼ ┌───────────┐ │ TRANSFORM │ ← Cleans, validates, enriches, flags anomalies └───────────┘ │ ▼ ┌───────────┐ │ LOAD │ ← PostgreSQL warehouse (momo_dw schema) └───────────┘ │ ▼ ┌───────────┐ │ dbt │ ← Analytical layer: 1 staging view + 4 mart tables └───────────┘ │ ▼ ┌───────────┐ │ Power BI │ ← 4-page live dashboard connected to PostgreSQL └───────────┘ │ ▼ ┌───────────┐ │ Kafka │ ← Real-time stream simulator: Producer + 3 Consumers └───────────┘ ``` --- ## ✅ What The Pipeline Does ### Extract - Reads raw MoMo transaction CSV files - Auto-generates 10,000 synthetic Ghana MoMo transactions if no file provided - Injects dirty records to demonstrate real data cleaning capability ### Transform - Removes duplicate and null transaction IDs - Parses and validates timestamps - Cleans and casts transaction amounts - Standardises text fields (operator, region, status) - Enriches with time dimensions (hour, day, month, quarter, weekend flag) - Categorises amounts into business buckets - Flags high-value transactions (top 5%) and rapid succession activity ### Load - Batch upserts into PostgreSQL (momo_dw schema) - Creates schema and tables automatically on first run - Auto-falls back to CSV export if database is unavailable - Logs every pipeline run to audit table --- ## 🔁 dbt Analyt …