KhalilBouslah/Tunisia_weather_streaming_pipeline
Domain:
climateRecord type:
softwareproject
Creator:
Kha
Host:
This project collects random weather data from Tunisian states, processes it in real-time, stores it in Cassandra, and visualizes it using Grafana.
# Weather_streaming
Weather Streaming Project
# Architecture Diagram
This project demonstrates a real-time weather data pipeline using the OpenWeather API, Kafka, Apache Spark, Cassandra, and Grafana. The pipeline collects random weather data from Tunisian states, processes it in real-time, stores it in Cassandra, and visualizes it using Grafana.
Project Overview
Technologies Used
OpenWeather API: For fetching weather data.
Kafka: For real-time data streaming.
Apache Spark: For processing streaming data.
Cassandra: For scalable, distributed data storage.
Grafana: For data visualization.
Pipeline Workflow
Producer: Fetches weather data from the OpenWeather API and sends it to a Kafka topic.
Stream Processing: Spark processes the streaming data from Kafka and stores it in Cassandra.
Storage & Export: Data stored in Cassandra is exported to a CSV file for further cleaning.
Visualization: Cleaned data is visualized in Grafana.
Project Setup
1. Initial Setup
Create a new project folder and navigate to it:
mkdir spark_project && cd spark_project
Create and activate a Python virtual environment:
python -m venv myenv
source myenv/bin/activate
2. Start Docker Containers
Use the docker-compose.yml file to pull and run necessary containers:
sudo docker-compose up -d
Running the Pipeline
1. Producer Script
Run the script to fetch weather data and send it to Kafka:
python producer_weather.py
2. Install Required Spark Packages
Download necessary packages for Kafka and Cassandra to connect with Spark:
# Run spark_stream.py:
spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10_2.12:3.5.4,com.datastax.spark:spark-cassandra-connector_2.12:3.5.1 spark_stream.py
Data Storage and Access
1. Access Cassandra
Run Cassandra's interactive shell:
sudo docker exec -it cassandra cqlsh -u cassandra -p cassandra localhost 9042
2. Describe keyspace spark_streams:
Describe spark_streams;
3. Query Weather Data
Check the captured events stored in Cass …