DIGI-UW/malawi-dhis2-pipeline

# Malawi DHIS2 HIV/TB Indicators Pipeline ## Overview This project implements a flexible, configuration-driven pipeline for importing HIV/TB health indicators from various Excel/CSV formats into DHIS2. Built on OpenFN and Instant OpenHIE v2, it supports multiple data sources and SFTP-based file uploads. ### Key Features - **Multi-Format Support**: Processes CSV/XLSX files with configuration-based column mapping - **Flexible Data Sources**: Google Sheets API and SFTP file monitoring - **Automated Processing**: Scheduled (cron) and event-driven (webhook) workflows - **Data Validation**: Built-in validation rules and transformation capabilities - **Time-Based Protection**: Configurable update windows to prevent accidental overwrites - **Docker Swarm Deployment**: Production-ready containerized architecture ## Quick Navigation ### For Operators (Deployment & Configuration) - **Quick Start Guide** - Deploy in 15 minutes - **Environment Setup** - Full installation guide - **Production Deployment** - Government instance setup - **Credential Configuration** - DHIS2 & SFTP credentials ### For Developers (Technical Reference) - **Technical Specifications** - SDD artifacts (spec, plan, tasks) - **Development Guide** - Workflow development - **OpenFN Design Patterns** - Best practices - **Testing Strategy** - Testing approach ## CI/CD The project includes automated CI testing via GitHub Actions: - **Environment Setup CI**: Tests the complete instant OpenHIE deployment - **Workflow Tests CI**: Validates OpenFN workflows using CLI testing framework View CI Documentation for details. ### Running CI Tests Locally Uses Docker to run GitHub Actions locally (no installation required): ```bash # Prerequisites: Docker must be running ./scripts/run-ci-locally.sh # Run all CI workflows ./scripts/run-ci-locally.sh --env-setup # Environment setup only ./scripts/run-ci-locally.sh --workflow-tests # Workflow tests only ./scripts/run-ci-locally.sh --list # Li …