fourlanes/ug-district-data-workflows

Data Orchestration for Uganda District Data # Uganda District Data Workflows A Prefect-based data cleaning pipeline for Uganda district data. The system processes diverse CSV datasets including facilities (health, education), demographic data, economic indicators, and other location-aggregated datasets while maintaining consistent location hierarchies and standardized outputs. ## Project Overview This pipeline handles three distinct data processing workflows: 1. **Facility Data Processing** - Individual records per facility with full location hierarchy, coordinate validation, and unique facility ID generation 2. **Aggregated Data Processing** - Summary data by location with validation and consistency checks 3. **Location Hierarchy Processing** - Extracts and standardizes location names, generates consistent location codes with unique hierarchical IDs ### Key Features - **Flexible Location Code Generation**: Hierarchical fallback system (village → parish → subcounty → district) ensures all records get appropriate location codes - **Cross-District Compatibility**: Dynamic processing flows handle multiple districts (Kayunga, Masindi) without modification - **Column Name Standardization**: Automatically handles both uppercase and lowercase column name variants - **Unique Hierarchical Location IDs**: URL-friendly IDs that prevent collisions (e.g., `d-kayunga`, `s-kayunga-kangulumira-town-council`) - **Smart Duplicate Prevention**: Updates existing facility records instead of creating duplicates - **Data Type Auto-Detection**: Automatically classifies datasets as facility vs aggregated data - **Thematic Area Mapping**: Identifies domain (health, education, water, etc.) from indicators - **Comprehensive Validation**: Location hierarchy validation, coordinate checks, and data quality reporting ## Setup ### Environment Setup ```bash python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate pip install -r requirements.txt ``` ## Running Workflows ### Facility Data Cleaning #### Single …