bachir00/west-africa-economic-data-pipeline

Pipeline ETL End-to-End (Airflow + AWS + PostgreSQL) # West Africa Economic Data Pipeline A production-grade ETL (Extract, Transform, Load) platform that automates the collection, processing, and warehousing of key economic indicators for 10 West African countries. Built with Apache Airflow, PostgreSQL, and AWS S3, this pipeline demonstrates enterprise-level data engineering practices including orchestration, error handling, data validation, and automated monitoring. ## 📋 Table of Contents - Overview - Architecture - Tech Stack - Features - Installation - Usage - Data Schema - Project Structure - Pipeline Workflows - Performance & Optimization - Error Handling & Monitoring - Testing - Future Enhancements - Contributing - License - Authors - Acknowledgments ## 📊 Overview This project ingests economic data from the **World Bank API** for 10 West African nations, applies rigorous data quality checks and transformations, and persists the results to both PostgreSQL (OLAP warehouse) and AWS S3 (data lake). The pipeline is fully orchestrated via Apache Airflow with retry logic, comprehensive logging, and data validation at multiple stages. **Key Metrics:** - **10 Countries**: Senegal, Nigeria, Ghana, Côte d'Ivoire, Mali, Burkina Faso, Guinea, Togo, Benin, Mauritania - **8 Economic Indicators**: GDP (current USD), Population, Inflation Rate, Unemployment Rate, Internet Users %, GDP per Capita, Digital Adoption Score, Economic Health Score - **Data Quality**: 13 dimensions tracked (ingestion timestamp, quality flags, validation status) - **Frequency**: Monthly automated runs via cron schedule - **Latency**: <5 minutes end-to-end (extract to warehouse) ## 🏗️ Architecture ``` ┌─────────────────────────────────────────────────────────────────────┐ │ ORCHESTRATION LAYER │ │ Apache Airflow (Scheduler) │ │ - DAG: west_africa_economic_pipeline │ │ - Schedule: @monthly …