aurora6174/airpulse-africa

Air Quality Data Engineering streaming pipeline for selected African Countries # AirPulse Africa > Real-time air quality monitoring across African cities — from sensor to dashboard. Air pollution is a leading cause of death in West Africa, yet real-time data remains inaccessible to most people. AirPulse Africa streams live PM2.5, NO2, PM10 and other pollutant readings from OpenAQ sensors across 5 African countries, processes them through a fully automated pipeline, and visualises WHO threshold breaches on an interactive dashboard. --- ## The Data Story Our pipeline revealed stark air quality realities across Africa: - **Nigeria (PM2.5)**: 64.7% of all readings exceed WHO safe limits — nearly 2 in every 3 hours - **Ethiopia (PM2.5)**: 100% breach rate — every single reading above the WHO threshold of 15 µg/m³ - **South Africa (PM10)**: Peak readings hit 180 µg/m³ — 4x the WHO limit of 45 µg/m³ - **Ghana (PM2.5)**: Average of 16.72 µg/m³ — just above the safe threshold - **Kenya**: Relatively cleaner air — 26.7% breach rate for PM2.5 --- ## Architecture --- ## Tech Stack | Layer | Tool | Purpose | |---|---|---| | Data source | OpenAQ v3 REST API | Real-time air quality sensor readings | | Message broker | Redpanda | Kafka-compatible streaming broker (Docker) | | Producer | Python + confluent-kafka | Polls OpenAQ, publishes to Redpanda topic | | Consumer 1 | Python + boto3 | Streams topic → AWS S3 data lake | | Consumer 2 | Python + snowflake-connector | Streams topic → Snowflake RAW table | | Cloud | AWS (S3 + IAM) | Data lake storage | | IaC | Terraform | Provisions S3 bucket, IAM user and policy | | Data warehouse | Snowflake | Clustered tables for analytics | | Transformations | dbt (dbt-snowflake) | Staging view + mart tables | | Orchestration | Prefect v2 | Schedules dbt runs every 5 minutes | | Dashboard | Streamlit | Interactive 2-tile dashboard | | Dev environment | WSL2 + Docker Engine + VS Code | Ubuntu on Windows | --- ## Dataset - **Source**: OpenAQ v3 API — free, open air quality data - **Countries**: Nigeria (NG), G …