25thOliver/Air-Quality-Pipeline

A fully containerized real-time data engineering project that monitors air quality for Kenyan cities (Nairobi & Mombasa) using Apache Kafka, MongoDB, Debezium, and Cassandra. It collects, streams, and analyzes hourly air pollution data (PM2.5, PM10, CO, NO₂, SO₂, Ozone, UV Index) — transforming raw sensor data into live, queryable insights for urb # Real-Time Air Quality Data Pipeline A complete end-to-end data engineering project that monitors air quality in real-time for Kenyan cities (Nairobi and Mombasa), demonstrating modern data pipeline architecture using streaming, change data capture (CDC), and analytics storage. --- ## Table of Contents 1. Project Overview 2. Architecture Overview 3. Technologies Used 4. Prerequisites 5. Quick Start 6. Understanding the Data Flow 7. Monitoring & Dashboards 8. Querying the Data 9. Troubleshooting 10. Future Enhancements --- ## Project Overview This project builds a **real-time data pipeline** that: - Collects air quality data (PM2.5, PM10, CO, NO2, SO2, Ozone, UV Index) every hour - Streams data through Kafka message queue - Stores raw data in MongoDB - Captures changes automatically using CDC (Change Data Capture) - Stores processed data in Cassandra for fast analytics **Key Features:** - Fully automated data collection (hourly schedule) - Real-time data streaming with Apache Kafka - Change Data Capture with Debezium - Multiple storage systems optimized for different use cases - Built-in monitoring dashboards - Production-ready error handling and retry logic - Fully containerized with Docker --- ## Architecture Overview ``` ┌─────────────────┐ │ Open-Meteo │ External Air Quality API │ API │ └────────┬────────┘ │ HTTP (hourly) ↓ ┌─────────────────┐ │ Producer │ Python script (data ingestion) └────────┬────────┘ │ Publish ↓ ┌─────────────────┐ │ Kafka Broker │ Message queue │ (air_quality_ │ │ data) │ └────────┬────────┘ │ Subscribe ↓ ┌─────────────────┐ │ Consumer │ MongoDB writer └────────┬────────┘ │ Insert ↓ ┌─────────────────┐ │ MongoDB │ Raw data storage (Replica Set) │ (air_quality_ │ │ raw) │ └────────┬────────┘ │ Monitor changes ↓ ┌─────────────────┐ │ Debezium │ Change Data Capture (CDC) │ Connector │ └────────┬────────┘ │ Publish changes ↓ ┌─────────────────┐ │ …