Chris-Ndungu/African-Market-Intelligence

# 🌍 African Market Intelligence Pipeline > **Automated macroeconomic data pipeline covering 54 African countries** — from raw API ingestion to query-optimised BigQuery analytics models, refreshed daily. Built to demonstrate production-grade data engineering: batch ingestion, dbt modelling, orchestration, and data quality — on a real-world African macroeconomic dataset. --- ## 🧩 The Problem Organisations operating in African markets — investors, development finance institutions, consultancies, and multinationals — face a persistent problem: macroeconomic data is **fragmented, inconsistently formatted, and manually intensive to gather**. | Pain Point | Current State | Impact | |---|---|---| | Fragmented data sources | World Bank, IMF, and national statistics portals each use different formats and access methods | Analysts spend 3–5 hours per country gathering baseline data | | No single source of truth | Exchange rate data in one spreadsheet, GDP in another, CPI in a third | Inconsistent figures across reports, version control failures | | Manual refresh cycles | Data updated manually when someone remembers — weekly at best | Decisions made on stale figures, no alerting when data goes out of date | | No data quality controls | No automated checks for missing values, outliers, or schema changes | Silent data quality failures go undetected until they surface in a report | This pipeline closes that gap. --- ## 🏗️ Architecture ``` ┌──────────────────────────────────────────────────────────────────┐ │ DATA SOURCES │ │ World Bank API · IMF Data API · ExchangeRate-API │ └────────────────────────┬─────────────────────────────────────────┘ │ Daily ingestion ▼ ┌──────────────────────────────────────────────────────────────────┐ │ INGESTION LAYER │ │ Python scripts (world_bank.py · imf.py · fx_rates.py) │ │ HTTP fetch → schema validati …