robgotherearly/ghana-export-monitor
Domaine:
socioeconomicType de record:
software
Créateur:
rob
Hôte:
Real-time monitoring of Ghana's export commodities (cocoa, gold, crude) and the cedi. Kafka -> Spark Structured Streaming -> Delta Lake / Postgres -> Grafana, on genuinely live free-tier data.
# Ghana Export Commodity & Cedi Monitoring Platform
Real-time data engineering on genuinely live, free data: cocoa, gold, crude and the
Ghanaian cedi, streamed through Kafka, processed with Spark, served from Postgres and
rendered on a Grafana dashboard that refreshes itself.
Cocoa and gold are Ghana's #1 and #2 exports and the cedi is live-traded — between them
they largely *are* the Ghanaian economy. Every source in this project is real and
free-tier; **the whole stack runs end to end without a single API key.**
```
docker compose up -d --build # or: make up
open localhost # Grafana, anonymous access enabled
```
---
## 1. Architecture
```mermaid
flowchart LR
subgraph ingest["Ingestion (Python)"]
P1[commodities poller]
P2[fx poller]
P3[weather poller]
end
subgraph bus["Apache Kafka (KRaft)"]
T1[(commodity.prices.raw)]
T2[(fx.rates.raw)]
T3[(weather.regions.raw)]
T4[(commodity.metrics)]
T5[(alerts.flagged)]
end
subgraph proc["Apache Spark"]
S1[Structured Streaming rolling windows + anomalies]
S2[Batch — nightly OHLC, seasonality, correlation]
end
L[(Delta Lake raw, partitioned by day)]
DB[(Postgres serving store)]
G[Grafana live dashboards]
N[alert notifier Slack / Telegram]
P1 --> T1
P2 --> T2
P3 --> T3
T1 & T2 & T3 --> S1
S1 --> L
S1 --> DB
S1 --> T4
S1 --> T5
T5 --> N
L --> S2
S2 --> DB
DB --> G
```
The streaming path and the batch path are genuinely separate: Spark Streaming writes
every raw tick to a Delta lake, and the nightly job recomputes the analytical tables
from *that lake*, never from Postgres and never by re-hitting an API. That is what makes
a replay or a backfill possible.
---
## 2. Live data sources
All defaults are key-free. Adding a key just moves a better provider to the front of the
chain — no code changes, no pipeline changes.
| Leg | Default (no key) | What it gives | Upgrades with a key |
|---|---|---|---|
| **Commodities** | Yahoo Finance chart API | Cocoa `CC=F` (ICE), gold `GC=F` (COMEX), WTI `CL=F` …