OluwasoreSamuel/My_open_weather_project

Weather updates for 36 cities in Nigeria using open weather Api using Airflow, s3 Buckets, Lambda, and redshift # 📗 My Open Weather Project **Weather updates for 36 cities in Nigeria using OpenWeather API with Airflow, AWS Lambda, S3 Buckets, and Redshift** Hello, This repository contains the code and resources for a data engineering project that collects weather data from the OpenWeather API and loads it through a fully automated ETL pipeline into AWS services for storage and analytics. The objective for this project is to use airflow to automate the ETL process using open weather api and transferring the data to aws s3 bucket.Also, using lambda to copy the collected data to a new bucket to prevent the main data from external damages with another lambda function to help reduce the unncessary data collected which is to be transferred to a data warehouse(Redshift). This project demonstrates practical data engineering skills including: 1. API ingestion 2. Cloud-native ETL pipelines 3. Serverless data processing 4. Workflow automation --- ## 📌 Table of Contents - Project Overview - Architecture - Tools & Technologies - Project Workflow - Repository Structure - Setup & Installation - Details by Component - Redshift Schema --- ## 📌 Project Overview This project implements an **end-to-end data engineering pipeline** that automatically extracts weather data for **36 Nigerian cities** using the **OpenWeather API**. The pipeline: - Extracts weather data from an external API - Stores raw data in Amazon S3 - Processes and cleans data using AWS Lambda - Loads transformed data into Amazon Redshift - Orchestrates tasks using Apache Airflow The processed data can later be used for analytics and visualization tools such as **Power BI**. --- ## 🧠 Architecture High-level architecture of the pipeline: OpenWeather API ↓ EC2 (Apache Airflow) ↓ Raw Weather CSV → S3 Bucket (Raw) ↓ AWS Lambda (File Transfer) ↓ S3 Bucket (Staging) ↓ AWS Lambda (Data Cleaning) ↓ S3 Bucket (Cleaned) ↓ Airflow → Amazon Redshift --- ## 🛠️ Tools & Technologies | Category | Technology | |--------|--- …