Lihle794/Employment-Data-Quality-Analytics-Pipeline-South-Africa-

A production-style data pipeline that ingests, validates, tracks and analyzes South African employment statistics while explicitly handling unreliable and changing data. # Employment Data Quality & Analytics Pipeline (South Africa) (github.com) ## Overview An end-to-end data engineering pipeline that ingests raw employment data, performs data quality validation, applies transformations and loads clean, analytics-ready data into a Dockerized PostgreSQL data warehouse. Built to reflect production-grade data engineering practices including layered warehouse architecture, automated testing, and CI/CD. ## Architecture ``` Raw CSV ↓ Python Ingestion (ingestion/ingest.py) ↓ Data Quality Validation (validation/quality_checks.py) ↓ Staging Schema (raw_employment) ↓ Analytics Schema (employment_metrics) ↓ Data Mart (province_summary) ``` ## Tech Stack - Python (pandas, psycopg2) - PostgreSQL 15 - Docker & Docker Compose - SQL (transformations & modeling) - pytest (automated testing) - GitHub Actions (CI/CD) ## Key Features - Data quality checks (nulls, ranges, schema validation) - Invalid record rejection with audit trail (rejected CSVs) - Province-level validation for all 9 South African provinces - Idempotent warehouse loading - Analytics schema with constraints - Load logging for observability - Containerized PostgreSQL warehouse - 17 automated tests covering ingestion and validation - CI/CD pipeline that runs tests and verifies Docker build on every push ## Project Structure ``` ├── .github/workflows/ # GitHub Actions CI/CD pipeline ├── ingestion/ # Raw data ingestion scripts ├── validation/ # Data quality checks & rejection logic ├── transformation/ # SQL transformations ├── analytics/ # Analytics queries and reporting ├── warehouse/ # Warehouse schema definitions ├── orchestration/ # Pipeline orchestration ├── docker/ # Docker Compose configuration ├── data/ │ ├── raw/ # Ingested raw files │ ├── processed/ # Clean, validated da …