Osei-dot/WeatherData_ETL_project

This project collect Weather Data for 6 West African Countries from OpenWeatherMap API, Transform and Load to Postgress and Update PowerBI dashboard with changes. The orchestration is achieved with Apache Airflow and the tasks will be run daily. Countries to be considered are Ghana, Benin, Nigeria, Ivory Coast, Togo and Guinea_Bissau **🌦️WEATHER DATA ETL FOR 6 AFRICAN CITIES.** This project demonstrates an **ETL (Extract, Transform, Load) pipeline** built with **Apache Airflow**. It fetches weather data from an API OpenWeathermapAPI, processes it, and stores it in PostgreSQL for downstream analysis. This project was done for 6 selected countries. 1. Ghana (Accra) 2. Nigeria (Abuja) 3. Benin (Cotonou) 4. Ivory Coast (Yamoussoukro) 5. Togo (Station Meteo) 6. Guinea (Kamsar) These countries/cities were purely selected at random for educational purposes. The pipeline is orchestrated using **Airflow DAGs** and can be extended to integrate with data warehouses or visualization tools (PowerBI was used) This can be used for any project that require Weather data from any of the listed project in a tabular form. With a raw json file format, the project cleans and transform them with an additional timestamp showing what day and time the weather was recorded. This is good enough to give you insight into how weatherData changes in a day. **Project Structure** Weather_ETL_Project/ │── **dags/** │ └── weather_etl_dag.py # Main Airflow DAG definition │ │── **Weather_ETL_Project/** │ └── main.py # Core ETL script (extract, transform, load) |── rawdata.json # Collect raw API queries for every API call |── etl_log.log # Collect logs filtered for logging.INFO |──error_log.log # Collect logs filtered for logging.ERROR │ │── requirements.txt # Python dependencies │── README.md # Project documentation **How to Use it** 1. Create and Activate Virtual Environment python3 -m venv airflow_env source airflow_env/bin/activate 3. Install Dependencies pip install -r requirements.txt 4. Initialize Airflow export AIRFLOW_HOME=~/airflow airflow db reset --yes airflow db migrate airflow users create \ --username admin \ --firstname First \ --lastname Last \ --role Admin \ --email admin@example.com 5. Start Services airflow scheduler & airflow webserver -p 8080 & The Airflow UI should now be avail …