Digital-Health-Agency/EVD-Warehouse-ETL

Kenya Survillance DataWarehouse ETL scripts # EVD Warehouse ETL Data warehouse ETL for EVD: MinIO → Postgres (`bronze` → `silver` → `gold`). ## Scope - **In scope**: MinIO → Postgres → dbt (silver/gold marts). - **Bronze is schema-inferred, not hand-modeled.** JSON records from each sending system are flattened and staged into `bronze.{source}_raw` tables whose columns are discovered from the data itself and evolve **additively** (new fields → new nullable columns; existing columns/data are never altered). - **No Data Vault modeling.** `silver`/`gold` are plain medallion-style dbt transforms — typing, dedup, joins — not hubs/links/satellites. See `docs/architecture.md` for the full design and `CLAUDE.md` for the conventions this repo follows. ## Stack | Layer | Tool | Notes | | ------------- | ------------ | ---------------------------------------------------------- | | Raw lake | MinIO | S3-compatible, bucket `evd`, one prefix per sending system | | Lake reader | DuckDB | `httpfs`, reads JSON (incl. gzip) directly from MinIO | | Orchestration | Dagster | Bronze ingestion assets + `@dbt_assets` wrapping dbt | | Warehouse | Postgres | Schemas: `bronze`, `silver`, `gold` | | Transform | dbt-postgres | Silver (typed/deduped) → gold (marts) | | Python deps | uv | Single `pyproject.toml` at root | ## Repository layout ``` EVD-Warehouse-ETL/ ├── infra/postgres/ # init.sql — CREATE SCHEMA bronze/silver/gold ├── orchestration/ │ └── evd_orchestration/ │ ├── resources/ # minio, duckdb_io, postgres │ ├── assets/ │ │ ├── bronze/ # schema inference + generic ingest asset factory │ │ └── transform/ # @dbt_assets wrapping transform/evd_transform │ ├── jobs.py, schedules.py, sensors.py │ └── tools/migrate.py # runs infra/postgres/init.sql ├── transform/ │ …