mazenelnaghy-code/egypt-air-quality

# Egypt Air Quality Warehouse An automated data pipeline that collects hourly air quality and weather readings for eight Egyptian cities, models them into a star schema, tests the result, and republishes a dashboard — every six hours, without a server. **Live dashboard:** mazenelnaghy-code.github.io --- ## What it does ``` Open-Meteo API │ ▼ EXTRACT ──────► data/raw/YYYY-MM-DD.jsonl append-only, source of truth │ ▼ TRANSFORM ─────► DuckDB warehouse staging → dims → facts → marts │ ▼ TEST ────────► 12 data quality assertions failures stop the run │ ▼ PUBLISH ──────► docs/ (dashboard + CSV exports) served by GitHub Pages ``` Orchestration is a GitHub Actions cron job. Public repositories get unlimited Actions minutes, so the whole thing runs indefinitely at no cost. --- ## Design decisions worth explaining **Raw data is never modified.** Extract writes what the API returned and nothing else. Every downstream table is derived. If a transform has a bug, fix it and rebuild — no data is lost. **Partitions are named for the day they describe**, not the day they were fetched, so a filename tells you what is inside it and "what happened on the 21st?" is one file rather than a scan of all of them. It also keeps a backfill proportionate: thirty days of history arrives as thirty ordinary files instead of one outsized file whose name claims a single day. A given date is appended to by every run whose window covers it — roughly four days' worth — and is then never touched again, so partitions go effectively immutable shortly after the fact. **The warehouse is rebuilt from scratch on every run.** At this volume that costs about a second, and it buys idempotency: any run produces exactly the same warehouse. A failed or half-finished run is never a problem, and backfilling is just another run. **Stored in UTC, reported in local time.** Storage is UTC because that is the only defensible choice for a timestamp. B …