mainamuragev/air_quality_pipeline

Real-time air quality pipeline for Nairobi & Mombasa using Kafka, Cassandra, and Python. Built for scalable ingestion, validation, and environmental insight. # Air Quality Pipeline Real-time air quality ingestion pipeline for Nairobi and Mombasa, built with Kafka, Cassandra, and Python. ## Tech Stack - **Kafka**: Stream ingestion - **Python**: ETL orchestration - **MongoDB**: Raw data storage - **Cassandra**: Scalable inserts - **Airflow**: Optional DAG scheduling ## Data Sources - Public environmental APIs - Custom ingestion scripts - Validated for Nairobi & Mombasa ## Features - Real-time ingestion and transformation - Scalable backend architecture - Reproducible workflows with clean logging - Modular design for future expansion ## Setup ```bash git clone github.com cd air_quality_pipeline python3 -m venv venv source venv/bin/activate pip install -r requirements.txt ``` ## Usage Run the ingestion script: ```bash python ingest_air_quality.py ``` ## Validation Includes: - URL checks - Schema enforcement - Logging for each stage ## Contributing Open to collaborators focused on environmental data, backend systems, or scalable infrastructure. ## Contact Maina Murage — muragevincent39@gmail.com