draletilouis/moh-pipeline

Production-ready ETL pipeline that transforms Uganda health sector Excel data (152 indicators, 72K+ records, 2016-2020) into a PostgreSQL star schema warehouse with a FastAPI REST API for analytics and BI integration. # Uganda Health Sector Data Pipeline > **Enterprise-grade ETL pipeline transforming Uganda's health sector performance data into actionable analytics** ## Table of Contents - Overview - Key Features - Architecture - Technology Stack - Quick Start - Project Structure - Data Pipeline - API Documentation - Authentication & Security - Observability & Monitoring - Data Model - Deployment - Testing - Troubleshooting - Contributing - License --- ## Overview A **production-ready data engineering platform** that transforms Uganda's health sector performance data from raw Excel files into a structured, queryable analytics system. This pipeline processes **72,402 health performance measurements** across **152 indicators**, spanning **2016/17 to 2019/20**, and serves them through a modern, secure REST API. **Scalable Data Ingestion**: Supports unlimited Excel files - simply drop multiple `.xlsx` or `.xls` files into the source directory, and the pipeline automatically discovers and processes them all. ### What Makes This Pipeline Enterprise-Grade? ✅ **Production Observability** - 151+ automated data quality checks, field-level lineage, and execution tracking ✅ **Security-First** - API key authentication, CORS configuration, input validation ✅ **Performance Optimized** - Star schema design, indexed queries, connection pooling ✅ **Multi-Source Support** - Automatic discovery and processing of multiple Excel files ✅ **Self-Documenting** - Comprehensive API docs, data dictionaries, inline documentation ✅ **Developer-Friendly** - CLI tools, interactive dashboards, detailed error messages ✅ **Battle-Tested** - Idempotent pipeline execution, error recovery, data validation --- ## Key Features ### Complete ETL Pipeline - **Extract** from multiple Excel files with automatic discovery and sheet detection - **Transform** with intelligent header detection and data unpivoting - **Load** to dimensional warehouse with UPSERT logic and integrity checks - **Scalable** - Suppor …