koomited/DataBeez_Hackaton_Data_Engineer

Building a pipeline to fetch daily temperature, meteo descripton, pressure, humidity for Senegal Cities: Dakar and Thies # Data Engineering Project --- --- ## Overview This project involves building a data pipeline to fetch and process daily weather data, including temperature, meteorological descriptions, pressure, and humidity, for Senegalese cities **Dakar** and **Thies**. The architecture is deployed using **Astronomer**, automated with **GitHub Actions**, and visualized in **Tableau** for insightful analysis. --- ## Features - **Weather Data Pipeline**: A robust ETL pipeline that extracts, transforms, and loads weather data into a PostgreSQL database. - **Automation**: Seamless CI/CD pipeline implemented with GitHub Actions for deployment and monitoring. - **Visualization**: Tableau dashboards to visualize weather trends and insights. - **Scalability**: Deployed with Astronomer to handle production-grade workloads. --- ## Project Contents This project contains the following files and folders: - **`dags/`**: Contains Airflow DAGs. - `etl_weather.py`: Processes weather data for a single API call. - `etl_senegal_weather.py`: Handles multiple API calls for Senegalese cities. - **`Dockerfile`**: Defines the Astronomer Runtime Docker image. - **`include/`**: Placeholder for additional resources. - **`packages.txt`**: Specifies OS-level dependencies (empty by default). - **`requirements.txt`**: Specifies Python dependencies (editable as per project needs). - **`plugins/`**: For custom or community plugins (empty by default). - **`airflow_settings.yaml`**: Configures Airflow Connections, Variables, and Pools locally. - **`.github/workflows/`**: Contains GitHub Actions workflows for CI/CD: - `create-deployment-preview.yml`: Creates a deployment preview. - `deploy-to-preview.yml`: Deploys code to the preview environment. - `delete-preview-deployment.yml`: Deletes the preview deployment. - `deploy-to-main-deployment.yml`: Deploys code to the main production environment. - **`images/`**: Contains the pipeline visualization diagram. --- ## Technologies Used 1. **A …