robgotherearly/ghana-export-monitor

Real-time monitoring of Ghana's export commodities (cocoa, gold, crude) and the cedi. Kafka -> Spark Structured Streaming -> Delta Lake / Postgres -> Grafana, on genuinely live free-tier data. # Ghana Export Commodity & Cedi Monitoring Platform Real-time data engineering on genuinely live, free data: cocoa, gold, crude and the Ghanaian cedi, streamed through Kafka, processed with Spark, served from Postgres and rendered on a Grafana dashboard that refreshes itself. Cocoa and gold are Ghana's #1 and #2 exports and the cedi is live-traded — between them they largely *are* the Ghanaian economy. Every source in this project is real and free-tier; **the whole stack runs end to end without a single API key.** ``` docker compose up -d --build # or: make up open localhost # Grafana, anonymous access enabled ``` --- ## 1. Architecture ```mermaid flowchart LR subgraph ingest["Ingestion (Python)"] P1[commodities poller] P2[fx poller] P3[weather poller] end subgraph bus["Apache Kafka (KRaft)"] T1[(commodity.prices.raw)] T2[(fx.rates.raw)] T3[(weather.regions.raw)] T4[(commodity.metrics)] T5[(alerts.flagged)] end subgraph proc["Apache Spark"] S1[Structured Streaming rolling windows + anomalies] S2[Batch — nightly OHLC, seasonality, correlation] end L[(Delta Lake raw, partitioned by day)] DB[(Postgres serving store)] G[Grafana live dashboards] N[alert notifier Slack / Telegram] P1 --> T1 P2 --> T2 P3 --> T3 T1 & T2 & T3 --> S1 S1 --> L S1 --> DB S1 --> T4 S1 --> T5 T5 --> N L --> S2 S2 --> DB DB --> G ``` The streaming path and the batch path are genuinely separate: Spark Streaming writes every raw tick to a Delta lake, and the nightly job recomputes the analytical tables from *that lake*, never from Postgres and never by re-hitting an API. That is what makes a replay or a backfill possible. --- ## 2. Live data sources All defaults are key-free. Adding a key just moves a better provider to the front of the chain — no code changes, no pipeline changes. | Leg | Default (no key) | What it gives | Upgrades with a key | |---|---|---|---| | **Commodities** | Yahoo Finance chart API | Cocoa `CC=F` (ICE), gold `GC=F` (COMEX), WTI `CL=F` …