Mouaad-BK/Weather_ETL_Pipeline

End-to-end ETL pipeline for collecting, processing, and visualizing hourly weather data from Eastern Morocco using Python, Airflow, PostgreSQL, and Power BI. # Weather ETL Pipeline An end-to-end ETL pipeline for collecting, transforming, storing, and analyzing hourly weather data, designed as a data engineering project focused on scalable climate data processing. ## Overview This project implements an automated ETL pipeline for collecting and processing hourly weather data from eight cities in Eastern Morocco: Oujda, Nador, Berkane, Driouch, Taourirt, Jerada, Guercif, and Figuig. Weather data is extracted from the OpenWeather API and enriched with static CSV files for older periods. The data is cleaned and transformed using Python scripts before being stored in a PostgreSQL database hosted on the Railway cloud platform. Apache Airflow orchestrates the workflow in a Docker environment, ensuring continuous hourly execution. The processed data is then used in Power BI to provide an interactive dashboard for climate analysis and decision support. The project demonstrates a scalable data engineering architecture designed for reliable weather data collection, historical storage, and analytical visualization. --- ## Architecture ## Features / Capabilities - Automated hourly extraction of weather data from the OpenWeather API - Integration of historical weather records from static CSV archives - Data cleaning, normalization, and preprocessing using Python - Validation checks to ensure consistency and reliability of records - Structured storage of hourly and cumulative historical datasets - Cloud-hosted PostgreSQL database for scalable data management - Automated workflow scheduling with Apache Airflow - Interactive Power BI dashboard for weather data visualization and analysis - Historical trend exploration and city-to-city comparison ## Tech Stack - Python - PostgreSQL - Railway (Cloud Hosting) - Apache Airflow - Docker - Power BI - OpenWeather API ## Project Structure ``` WEATHER_ETL_PIPELINE/ │ ├── airflow/ │ ├── dags/ # Airflow DAG definitions │ ├── logs/ # Airflow execution logs │ ├─ …