wizli595/morocco-prices

Morocco Price Observatory — enterprise data pipeline for historical and current price data MaPrix — Morocco Price Observatory An enterprise-grade data platform that collects, processes, and serves historical and current price data for Morocco — from a loaf of *khobz* to a head of sheep. ## Why This Exists No unified dataset exists for Moroccan prices. Data is scattered across 30+ sources in 3 languages — behind APIs, in Excel files, in PDFs, in news articles. This project unifies it all into a single, queryable, versioned dataset. ## What's Inside - **500+ products**: Food, energy, housing, livestock, services, agricultural exports - **Historical depth**: CPI from the 1960s, producer prices from 1991, market prices from 2010 - **Regional granularity**: National → 12 regions → 75 provinces → 18+ cities → markets - **10+ data sources**: World Bank, FAOSTAT, HCP, WFP, news media, manual archives - **Enterprise pipeline**: Kafka → Spark → Delta Lake → PostgreSQL → API/Superset ## Project Status Early build (Phases 0–4). The **script-based ingestion path is working and validated end-to-end**: | Stage | State | |-------|-------| | Domain core (models, ports, transformers, registry, chain) | ✅ built, 50 tests | | Collectors — World Bank, FAOSTAT, Manual | ✅ live, plugin-registered | | Warehouse ingestion → PostgreSQL star schema | ✅ working (`run_collectors` → `run_processing`) | | Enrichment — unit normalization, historical FX, inflation | ✅ working | | Kafka → Spark → Delta medallion layers | 🚧 infra defined, not yet wired end-to-end | | Quality (Great Expectations), API routes, Superset, Airflow | ⏳ planned | **Validated:** the live FAOSTAT sheep producer-price series (1991–1998) reproduces the documented figures exactly after unit + currency normalization (e.g. 1998 → 51.25 MAD/kg). See `docs/sources/sheep_prices.md` and `docs/sources/price_research_2026.md`. ## Architecture ``` Collectors → Kafka → Spark → Delta Lake (Bronze/Silver/Gold) → PostgreSQL → API + Superset ↓ Grafana (Metrics + Logs) ``` Three patterns, each solving a distinct …