kevinakhondo/African-Intelligence-Platform

# African Market Intelligence Platform An end-to-end data engineering platform that ingests real-time African market data into Snowflake, transforms it with dbt, orchestrates pipelines with Airflow, and visualises insights in Metabase. ## Architecture APIs → Python ingestion → Snowflake (RAW → STAGING → MARTS) → Metabase ## Stack | Layer | Tool | |---|---| | Ingestion | Python 3 | | Warehouse | Snowflake | | Transformation | dbt | | Orchestration | Apache Airflow | | BI | Metabase | | Containerisation | Docker | | CI/CD | GitHub Actions | ## Data Sources | Data | Source | Refresh | |---|---|---| | FX Rates | exchangerate-api.com | Every 15 min | | Crypto Prices | CoinGecko API | Every 15 min | | News Sentiment | NewsAPI + TextBlob | Hourly | | Job Market | RemoteOK API | Daily | ## Setup 1. Clone the repo 2. Copy `.env.example` to `.env` and fill in your credentials 3. Run `source venv/bin/activate` to activate Python environment 4. Run `pip install -r requirements.txt` 5. Run `docker compose up` to start Airflow 6. Run `dbt run` to execute transformations ## Project Status - [x] Phase 1: Project setup and structure - [ ] Phase 2: Python ingestion scripts - [ ] Phase 3: Snowflake warehouse setup - [ ] Phase 4: dbt transformations - [ ] Phase 5: Airflow orchestration - [ ] Phase 6: Metabase dashboards - [ ] Phase 7: CI/CD and documentation ## Author Kevin Akhondo — Data Engineer, Nairobi, Kenya