KhalilBouslah/Tunisia_weather_streaming_pipeline

This project collects random weather data from Tunisian states, processes it in real-time, stores it in Cassandra, and visualizes it using Grafana. # Weather_streaming Weather Streaming Project # Architecture Diagram This project demonstrates a real-time weather data pipeline using the OpenWeather API, Kafka, Apache Spark, Cassandra, and Grafana. The pipeline collects random weather data from Tunisian states, processes it in real-time, stores it in Cassandra, and visualizes it using Grafana. Project Overview Technologies Used OpenWeather API: For fetching weather data. Kafka: For real-time data streaming. Apache Spark: For processing streaming data. Cassandra: For scalable, distributed data storage. Grafana: For data visualization. Pipeline Workflow Producer: Fetches weather data from the OpenWeather API and sends it to a Kafka topic. Stream Processing: Spark processes the streaming data from Kafka and stores it in Cassandra. Storage & Export: Data stored in Cassandra is exported to a CSV file for further cleaning. Visualization: Cleaned data is visualized in Grafana. Project Setup 1. Initial Setup Create a new project folder and navigate to it: mkdir spark_project && cd spark_project Create and activate a Python virtual environment: python -m venv myenv source myenv/bin/activate 2. Start Docker Containers Use the docker-compose.yml file to pull and run necessary containers: sudo docker-compose up -d Running the Pipeline 1. Producer Script Run the script to fetch weather data and send it to Kafka: python producer_weather.py 2. Install Required Spark Packages Download necessary packages for Kafka and Cassandra to connect with Spark: # Run spark_stream.py: spark-submit --packages org.apache.spark:spark-streaming-kafka-0-10_2.12:3.5.4,com.datastax.spark:spark-cassandra-connector_2.12:3.5.1 spark_stream.py Data Storage and Access 1. Access Cassandra Run Cassandra's interactive shell: sudo docker exec -it cassandra cqlsh -u cassandra -p cassandra localhost 9042 2. Describe keyspace spark_streams: Describe spark_streams; 3. Query Weather Data Check the captured events stored in Cass …