Pysham0n/Weather-Streaming-Spark

This project is a real-time weather monitoring application that collects weather data from various cities in Tunisia (Tunis, Sfax, Mahdia) using big data technologies. # Streaming Weather Monitoring Application ## Overview This project is a real-time weather monitoring application that collects weather data from various cities in Tunisia (Tunis, Sfax, Mahdia) and streams it to Kafka topics. It includes components for data collection, processing, and visualization. ## Description The project aims to: - Collect real-time weather data from multiple locations using the Tomorrow.io API. - Stream weather data to Kafka topics for real-time processing. - Use Apache Spark for processing and analyzing the weather data. - Provide a RESTful API for accessing weather data using Flask RestX. - Display a real-time weather dashboard using Dash and Plotly. ## Pipeline Here you can find a detailed overview of our application pipeline: ## Prerequisites In order to run this projet you need to have: 3 or more working virtual machines with linux os installed on each one of them (i used lubuntu os). Fully configured and installed hadoop in cluster mode (i used hadoop 3.3.4). Apache Spark installed on top of hadoop (i used spark 3.4.1). Fully configured and installed Apache Kafka in cluster mode (i used kafka 3.6.1). Apache Hive installed (i used hive 3.1.3). Apache Airflow installed (version >=2.0.0). Python 3 or later installed. Java 8 installed (it's the only java version supported by hive). ## Setup and Usage To set up the project locally, follow these steps: 1) Clone the repository: > git clone 2) Install dependencies: > pip install -r requirements.txt 3) Start your hadoop cluster. 4) start hive and create weather table using this command(modify location to match your hdfs environment): > CREATE EXTERNAL TABLE weather_details_tb1 ( CityName STRING, Temperature DOUBLE, Humidity DOUBLE, RainIntensity INTEGER, WindSpeed DOUBLE, WindDirection DOUBLE, CreationTime TIMESTAMP) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' STORED AS TEXTFILE LOCATION '/user/hadoop/outputs/spark_outputs/weatherstreaming/data'; 5) Create an …