jtambe007/cameroon-data-pipeline

An enterprise-grade, automated ELT (Extract, Load, Transform) data pipeline engineered to ingest, clean, validate, and model fragmented public-sector macroeconomic indicators and regional infrastructure development metrics into an analytical warehouse layer. # 🌍 Cameroon Macro-Economic & Infrastructure Data Pipeline Status: In-Progress An enterprise-grade, automated ELT (Extract, Load, Transform) data pipeline engineered to ingest, clean, validate, and model fragmented public-sector macroeconomic indicators and regional infrastructure development metrics into an analytical warehouse layer. ### 💼 Why Agencies & Enterprise Buyers Care: * **Handling High-Volatility Data:** Public-sector and international development data are notoriously messy, unstandardized, and prone to sudden schema drifts. This pipeline serves as a blueprint for handling unpredictable multi-source APIs and web-scraping pipelines. * **Production-Grade Infrastructure on a Zero-Dollar Budget:** Demonstrates how to leverage localized analytics engineering tooling (**DuckDB + dbt Core + GitHub Actions**) to build high-performance data platforms without incurring premature cloud warehouse overhead costs. --- ## 🏗️ Target Architecture Flow ### 1. Ingestion & Extraction Layer (EL) * **Dynamic Scraping & API Ingestion:** Modular Python scripts running asynchronous network requests to harvest multi-source public datasets (economic reports, infrastructure registries, parsing unformatted PDF/HTML tables). * **Resiliency Protocols:** Built-in error-handling, request throttling, and retry mechanisms to maintain data collection continuity despite volatile public endpoint uptimes. ### 2. Analytical Storage Layer (L) * **DuckDB Data Lakehouse:** Utilizing local transactional DuckDB files for highly localized, lightning-fast columnar execution. * **Cloud Expansion Path:** Engineered with explicit decoupling, allowing seamless migration to **Google BigQuery** or **Snowflake** warehouses via minor dbt profile environment adjustments. ### 3. Transformation & Modeling Layer (T) * **dbt Core Framework:** Translating raw staging tables into highly structural, optimized dimensional star-schemas (`fct_` and `dim_` tables) optimized for business intelligence de …