philwem/african-air-traffic-databricks
Domain:
mobilityRecord type:
project
Creator:
phi
Host:
# African Air Traffic Analytics — End-to-End Data Engineering Pipeline
## Overview
A production-grade data engineering pipeline that ingests, transforms, and analyses
real-time African air traffic data. Built on Databricks with Delta Lake, PySpark, and
a Medallion Architecture (Bronze → Silver → Gold), this project demonstrates end-to-end
data pipeline design, data quality enforcement, and scalable analytics on live aviation data.
**Research question:** *What patterns exist in African air traffic — which airlines dominate,
which routes are busiest, and how does traffic vary by region and time of day?*
---
## Architecture
```
Data Sources
│
├── OpenSky Network API (live flight positions — African bounding box)
├── ADSBDB API (aircraft + airline metadata)
└── OurAirports CSV (airport reference data — 84,000+ airports)
│
▼
┌─────────────────────────────────────────────────────┐
│ BRONZE LAYER (Delta Lake — raw, append-only) │
│ flight_states │ aircraft_meta │ airline_meta │
│ airports │
└─────────────────────────────────────────────────────┘
│ PySpark cleaning + deduplication
▼
┌─────────────────────────────────────────────────────┐
│ SILVER LAYER (Delta Lake — cleaned, typed) │
│ flight_positions │ dim_aircraft │ dim_airline │
│ dim_airport │
└─────────────────────────────────────────────────────┘
│ Spark SQL incremental aggregations
▼
┌─────────────────────────────────────────────────────┐
│ GOLD LAYER (Delta Lake — analytics-ready) │
│ mart_airline_kpis_daily │
│ mart_aircraft_activity_daily │
└─────────────────────────────────────────────────────┘
│
▼
Databricks SQL │ Power BI │ Analysis Notebooks
```
---
## Tech Stack
| Layer | Technology | Purpose |
|---|---|---|
| Ingestion | PySpark + requests | API calls, schema enforcement |
| Storage | Delta La …