LubangaD/lorowandatapipiline

Data pipeline for LoRowan weather devices in Kenya # IMARIKA Weather Data Processing Pipeline A real-time weather data processing pipeline built with Apache Spark, Kafka, and PostgreSQL. This system fetches weather data from an external API, processes it through a streaming pipeline with data cleaning, imputation, and anomaly detection, then stores both raw and processed data in PostgreSQL. ## 🏗️ Architecture ``` Weather API → Kafka Producer → Kafka Topic → Spark Streaming → PostgreSQL ↓ [Data Processing Pipeline] • Data Cleaning • ML Imputation • Anomaly Detection • Daily Aggregation +-------------------+ | Weather API | +--------+----------+ | v +--------+----------+ | Kafka Producer | | (api_data_fetcher)| [Pushes data to Kafka Topic] +--------+----------+ | v +-------------------+ | Kafka Topic | | (weather-readings)| +--------+----------+ | v +-----------------------------+ | Spark Streaming | | - Data Cleaning | | - ML Imputation | | - Anomaly Detection | | - Aggregation | +--------+----------+--------+ | | +--------v--+ +-------v--------+ | Raw Table | | Processed Table| | PostgreSQL| | PostgreSQL | +--------+--+ +-------+--------+ | | | +---------v----------+ +-------->+ Visualization (BI) | | - Power BI | +--------------------+ +---------------------+ | Forecasting Layer | | - Neural GCM | +---------------------+ ``` ## 📊 Features - **Real-time Data Ingestion**: Fetches weather data from external API every 3 hours - **Stream Processing**: Apache Spark Structured Streaming for real-time data processing - **Data Quality**: Comprehensive data cleaning and validation - **ML Pipeline**: Mean-based imputation and Z-score anomaly detection - **Data Aggregation**: Daily weather summaries and statistics - **Monitoring**: Kafka UI (Kafdrop) for stream monitoring - **Containerized**: Fully dockerized environment for easy deployment ## 🛠️ Tech St …