Thiyane24/etl-pipeline-moz-economy

End-to-end ETL pipeline using medallion architecture (Bronze/Silver/Gold) on Mozambique World Bank economic data # ETL Pipeline: Mozambique Economic Data **Looking for a custom ETL pipeline for your business? Hire me on Fiverr** ## Overview This is an ETL (Extract, Transform, Load) pipeline that processes Mozambique's economic data from the World Bank. The pipeline extracts data from a CSV file, cleans and standardizes it, loads it into a database, and answers business questions about the data. ## Pipeline Stages ### 1. Extract (Bronze Layer) - Reads messy CSV file using pandas - Adds audit columns: `_row_hash`, `_load_timestamp`, `_load_id` - Preserves raw data exactly as received - **Output:** Parquet files in `Data/bronze_raw/` ### 2. Transform (Silver Layer) - Removes rows with missing critical values - Standardizes column formatting (uppercase country codes, title case names) - Replaces inconsistencies in data values - Extracts and standardizes year values from various formats - Removes irrelevant columns - **Output:** Cleaned Parquet files in `Data/silver/` ### 3. Load - Loads cleaned data to SQLite database (`database/economy.db`) - Stores data in `economic_data` table for querying ### 4. Gold Layer (Business Insights) - Answers business questions using SQL queries - **BQ1:** GDP growth rate year-over-year (1982-2024) - **BQ4:** Data completeness audit (which indicators have complete data) - **Output:** Results exported as CSV and Parquet files in `Data/gold/` ## Architecture ``` Bronze Layer (Raw) ↓ Silver Layer (Clean) ↓ Load to Database ↓ Gold Layer (Insights) ``` The medallion architecture separates data into three states: - **Bronze:** Raw, untouched data (audit trail) - **Silver:** Cleaned, standardized data (ready for analysis) - **Gold:** Business-ready insights and answers ## How to Run ### Build the Docker Image ```bash docker build -t etl-pipeline-economy . ``` ### Run the Pipeline ```bash docker run -v $(pwd)/Data:/app/Data -v $(pwd)/database:/app/database etl-pipeline-economy ``` This will: 1. Extract raw CSV → Bronze Parquet files 2. Clean …