MEDICONSORTIUM/eo-datapipeline

Automated pipeline for extracting Africa malaria risk data from Google Earth Engine and ingesting it into AWS RDS. EOHA EO Data Pipeline Automated monthly pipeline that extracts satellite and reanalysis data from Google Earth Engine for all South African provinces and ingests per-ward environmental indicators into an AWS RDS PostgreSQL database — powering the EOHA (Earth Observational Health Analytics) malaria risk platform. **MED Consortium · EOHA Platform · PI: Kagisho Montjane** *Funded by TuksNovation and NeoFrontiers* --- ## 📋 Table of Contents 1. Introduction 2. Tech Stack 3. Pipeline Architecture 4. Repository Structure 5. Data Sources 6. Database Schema 7. Quick Start 8. Running the Tests 9. Environment Variables 10. Deployment 11. Contributing --- ## 🤖 Introduction The EOHA EO Data Pipeline is a scheduled batch job that runs once per month to ingest satellite-derived environmental risk factors into a PostgreSQL database. These indicators — land surface temperature, vegetation index, water index, soil moisture, population density, and land cover — are used downstream by the EOHA platform to model malaria transmission risk at ward level across South Africa. The pipeline processes **all 9 South African provinces** in parallel batch export tasks on Google Earth Engine, then sequentially downloads and ingests each province's ward-level CSV into the database. The entire process is idempotent: re-running the pipeline for the same month produces the same database state. **Key design decisions:** - Business logic is extracted into testable pure functions in `eo-datapipeline.py`. The original `etl_pipeline.py` script remains the production entry point but delegates all transform logic to the module. - All 98 automated tests run **without any GCP credentials** — external surfaces are mocked at the module boundary. - Every PostgreSQL write uses `ON CONFLICT DO UPDATE` to ensure safe re-runs. --- ## ⚙️ Tech Stack | Layer | Technology | |-------|-----------| | Satellite data | Google Earth Engine (MODIS, ERA5, WorldPop, ESA WorldCover) | | Cloud storage | Google C …