yoel3imari/morocco-wildfire-pipeline
Domaine:
environment and energygeospatialType de record:
software
Créateur:
yoe
Hôte:
Data engineering platform computing fire risk grids, DBSCAN fire-event clusters, and infrastructure exposure for Morocco — from raw NASA FIRMS / ERA5 / OSM data through a Bronze→Silver→Gold medallion warehouse (Airflow, MinIO, GeoParquet, PostGIS, dbt + DuckDB), served on a MapLibre web map.
# Wildfire Intelligence & Risk Monitoring Platform (Morocco)
A production-style cloud-native geospatial data platform that continuously ingests, cleanses, enriches, and visualizes wildfire intelligence across **Morocco** from **NASA FIRMS** satellite detections, **Open-Meteo ERA5** weather reanalysis, and **OpenStreetMap** infrastructure data.
Built around a **Bronze $\rightarrow$ Silver $\rightarrow$ Gold Medallion Architecture**, orchestrated by **Apache Airflow**, stored in **MinIO (S3)** and **PostGIS**, transformed with **dbt** & **DuckDB**, and served via a **FastAPI** web map and REST API.
---
## 🏛️ Architecture & Medallion Data Flow
```mermaid
flowchart TD
subgraph Data Sources
FIRMS["NASA FIRMS API (VIIRS & MODIS NRT CSV)"]
METEO["Open-Meteo API (ERA5 Weather JSON)"]
OSM["GeoFabrik OSM (Morocco PBF Extract)"]
end
subgraph Airflow Orchestration
DAG1["firms_ingest DAG"]
DAG2["weather_ingest DAG"]
DAG3["osm_ingest DAG"]
DAG4["firms_silver DAG"]
DAG5["weather_silver DAG"]
end
subgraph Bronze Layer [MinIO S3 - Raw Data]
B_FIRMS["bronze/firms/{source}/{date}/*.csv"]
B_METEO["bronze/weather/{date}/raw.json"]
B_OSM["bronze/osm/morocco-latest.osm.pbf"]
end
subgraph Silver Layer [MinIO S3 - Cleaned GeoParquet]
S_FIRMS["silver/firms/{source}/{date}/*.parquet (Point WGS84, BBox Filtered)"]
S_METEO["silver/weather/{date}/*.parquet (Normalized Hourly Tidy Data)"]
QUAL["Data Quality Checks (Completeness, Schema, Coords)"]
end
subgraph Gold Layer [PostGIS 16 Warehouse via dbt & DuckDB]
osm2pg["osm2pgsql Import"]
F_DET["fact_fire_detection (Granular Satellite Detections)"]
D_EVT["dim_fire_event (DBSCAN Spatial-Temporal Clustering)"]
RISK["fire_risk_grid (0.1° Grid: Wind, Humidity, Dryness)"]
EXPOS["infrastructure_exposure (KNN Distance to Cities, Roads, Hospitals)"]
end
FIRMS --> DAG1 --> B_FIRMS
METEO --> DAG2 --> B_METEO
OSM --> DAG3 --> B_OSM
B_FIRMS --> DAG4 --> S_FIRMS --> QUAL
B_METEO --> DAG5 --> S_METEO --> QUAL
S_FIRMS --> F_DET
S_METEO --> RISK
B_OSM - …