manasseh-asaah/etl-data-pipeline

Modular ETL pipeline: extracts African country data from a REST API, transforms and validates it, loads into SQLite and runs analytical SQL queries # 🔄 African Countries ETL Pipeline A modular **Extract → Transform → Load (ETL)** pipeline that ingests live African country data from a public REST API, applies data quality transformations, loads the result into a **SQLite** database, and runs analytical **SQL** queries — all in a single command. ## 🏗️ Pipeline Architecture ``` REST Countries API │ ▼ [1] EXTRACT fetch_countries() — HTTP GET, error handling │ ▼ [2] TRANSFORM transform() — flatten JSON, compute density, cast types │ ▼ [3] VALIDATE validate() — data quality checks (nulls, types, ranges) │ ▼ [4] LOAD load_to_sqlite() — full-refresh into SQLite │ ▼ [5] QUERY run_queries() — 5 analytical SQL queries │ ▼ [6] EXPORT export_csv() — CSV for downstream use ``` ## 🛠️ Tech Stack | Tool | Purpose | |------|---------| | Python 3.10+ | Core language | | `requests` | API extraction | | `pandas` | Data transformation & quality checks | | `sqlite3` | Embedded database (stdlib) | | SQL | Analytical querying | ## 🚀 Getting Started ```bash # Clone the repo git clone github.com cd etl-data-pipeline # Install dependencies pip install -r requirements.txt # Run the pipeline python main.py ``` ## 📁 Project Structure ``` etl-data-pipeline/ ├── main.py # Orchestrator — runs all 5 stages ├── config.py # Centralised configuration ├── requirements.txt ├── pipeline/ │ ├── extract.py # Stage 1: API extraction │ ├── transform.py # Stage 2 & 3: transformation + validation │ └── load.py # Stage 4–6: loading, querying, export ├── data/ # SQLite database (auto-created) └── outputs/ # CSV export + pipeline log (auto-created) ``` ## 📊 Analytical SQL Queries | Query | Description | |-------|-------------| | `top_10_by_population` | Most populous African countries | | `top_10_by_area` | Largest countries by land area | | `hig …