yoel3imari/morocco-wildfire-pipeline

Data engineering platform computing fire risk grids, DBSCAN fire-event clusters, and infrastructure exposure for Morocco — from raw NASA FIRMS / ERA5 / OSM data through a Bronze→Silver→Gold medallion warehouse (Airflow, MinIO, GeoParquet, PostGIS, dbt + DuckDB), served on a MapLibre web map. # Wildfire Intelligence & Risk Monitoring Platform (Morocco) A production-style cloud-native geospatial data platform that continuously ingests, cleanses, enriches, and visualizes wildfire intelligence across **Morocco** from **NASA FIRMS** satellite detections, **Open-Meteo ERA5** weather reanalysis, and **OpenStreetMap** infrastructure data. Built around a **Bronze $\rightarrow$ Silver $\rightarrow$ Gold Medallion Architecture**, orchestrated by **Apache Airflow**, stored in **MinIO (S3)** and **PostGIS**, transformed with **dbt** & **DuckDB**, and served via a **FastAPI** web map and REST API. --- ## 🏛️ Architecture & Medallion Data Flow ```mermaid flowchart TD subgraph Data Sources FIRMS["NASA FIRMS API (VIIRS & MODIS NRT CSV)"] METEO["Open-Meteo API (ERA5 Weather JSON)"] OSM["GeoFabrik OSM (Morocco PBF Extract)"] end subgraph Airflow Orchestration DAG1["firms_ingest DAG"] DAG2["weather_ingest DAG"] DAG3["osm_ingest DAG"] DAG4["firms_silver DAG"] DAG5["weather_silver DAG"] end subgraph Bronze Layer [MinIO S3 - Raw Data] B_FIRMS["bronze/firms/{source}/{date}/*.csv"] B_METEO["bronze/weather/{date}/raw.json"] B_OSM["bronze/osm/morocco-latest.osm.pbf"] end subgraph Silver Layer [MinIO S3 - Cleaned GeoParquet] S_FIRMS["silver/firms/{source}/{date}/*.parquet (Point WGS84, BBox Filtered)"] S_METEO["silver/weather/{date}/*.parquet (Normalized Hourly Tidy Data)"] QUAL["Data Quality Checks (Completeness, Schema, Coords)"] end subgraph Gold Layer [PostGIS 16 Warehouse via dbt & DuckDB] osm2pg["osm2pgsql Import"] F_DET["fact_fire_detection (Granular Satellite Detections)"] D_EVT["dim_fire_event (DBSCAN Spatial-Temporal Clustering)"] RISK["fire_risk_grid (0.1° Grid: Wind, Humidity, Dryness)"] EXPOS["infrastructure_exposure (KNN Distance to Cities, Roads, Hospitals)"] end FIRMS --> DAG1 --> B_FIRMS METEO --> DAG2 --> B_METEO OSM --> DAG3 --> B_OSM B_FIRMS --> DAG4 --> S_FIRMS --> QUAL B_METEO --> DAG5 --> S_METEO --> QUAL S_FIRMS --> F_DET S_METEO --> RISK B_OSM - …