draletilouis/moh-pipeline
Domaine:
healthcareType de record:
software
Créateur:
dra
Hôte:
Production-ready ETL pipeline that transforms Uganda health sector Excel data (152 indicators, 72K+ records, 2016-2020) into a PostgreSQL star schema warehouse with a FastAPI REST API for analytics and BI integration.
# Uganda Health Sector Data Pipeline
> **Enterprise-grade ETL pipeline transforming Uganda's health sector performance data into actionable analytics**
## Table of Contents
- Overview
- Key Features
- Architecture
- Technology Stack
- Quick Start
- Project Structure
- Data Pipeline
- API Documentation
- Authentication & Security
- Observability & Monitoring
- Data Model
- Deployment
- Testing
- Troubleshooting
- Contributing
- License
---
## Overview
A **production-ready data engineering platform** that transforms Uganda's health sector performance data from raw Excel files into a structured, queryable analytics system. This pipeline processes **72,402 health performance measurements** across **152 indicators**, spanning **2016/17 to 2019/20**, and serves them through a modern, secure REST API.
**Scalable Data Ingestion**: Supports unlimited Excel files - simply drop multiple `.xlsx` or `.xls` files into the source directory, and the pipeline automatically discovers and processes them all.
### What Makes This Pipeline Enterprise-Grade?
✅ **Production Observability** - 151+ automated data quality checks, field-level lineage, and execution tracking
✅ **Security-First** - API key authentication, CORS configuration, input validation
✅ **Performance Optimized** - Star schema design, indexed queries, connection pooling
✅ **Multi-Source Support** - Automatic discovery and processing of multiple Excel files
✅ **Self-Documenting** - Comprehensive API docs, data dictionaries, inline documentation
✅ **Developer-Friendly** - CLI tools, interactive dashboards, detailed error messages
✅ **Battle-Tested** - Idempotent pipeline execution, error recovery, data validation
---
## Key Features
### Complete ETL Pipeline
- **Extract** from multiple Excel files with automatic discovery and sheet detection
- **Transform** with intelligent header detection and data unpivoting
- **Load** to dimensional warehouse with UPSERT logic and integrity checks
- **Scalable** - Suppor …