Benness001/weather-etl-pipeline

Automated hourly ETL pipeline for Lagos, Nigeria weather data — Python, SQL, Apache Airflow, Snowflake & Databricks # 🌤️Lagos Weather ETL Pipeline An automated data engineering pipeline that extracts real-time weather data for Lagos, Nigeria, transforms it into a structured format, and loads it into Snowflake — orchestrated hourly with Apache Airflow. --- ## Architecture OpenWeatherMap API → Python ETL → Apache Airflow → Snowflake --- ## 🛠️ Tech Stack | Layer | Tool | |----------------|-------------------------| | Data Source | OpenWeatherMap API | | Language | Python 3.12 | | Orchestration | Apache Airflow 2.9.1 | | Data Warehouse | Snowflake | | Environment | Ubuntu/WSL | --- ## 📁 Project Structure ``` weather_pipeline/ ├── airflow/ │ └── dags/ │ └── weather_dag.py # Airflow DAG — orchestrates ETL hourly ├── notebooks/ │ └── weather_etl_databricks.py # Databricks notebook version ├── scripts/ │ ├── __init__.py │ ├── extract.py # Pulls data from OpenWeatherMap API │ ├── transform.py # Cleans and structures raw data │ └── load.py # Loads data into Snowflake ├── snowflake/ │ └── weather_table.sql # Snowflake table setup script ├── .env.example # Credentials template ├── .gitignore # Ignores .env and sensitive files ├── requirements.txt # Python dependencies ├── test_pipeline.py # Manual end-to-end test script └── README.md ``` --- ## ⚙️ Setup Instructions ### 1. Clone the repo ```bash git clone github.com cd weather-pipeline ``` ### 2. Create and activate virtual environment ```bash python3 -m venv weather_venv source weather_venv/bin/activate ``` ### 3. Install dependencies ```bash pip install -r requirements.txt ``` ### 4. Create your `.env` file ```bash cp .env.example .env # Fill in your API keys and Snowflake credentials ``` ### 5. Set up Snowflake table Run the SQL in ` …