declerke/Kenya-Real-Estate-Pipeline

# 🏘️ Kenya Real Estate Multi-Source Pipeline: Cross-Platform Property Intelligence for the Kenyan Market **Kenya Real Estate Pipeline** is a production-grade data engineering project that aggregates, normalises, and analyses property listings from Kenya's two largest real estate platforms — **Pigiame** and **BuyRentKenya** — into a unified PostgreSQL warehouse with a dbt transformation layer and Apache Superset dashboards. It scrapes 11 property categories daily, deduplicates across sources, computes price-per-sqm metrics, and surfaces cross-source price comparisons by county, neighbourhood, property type, and bedroom count — all orchestrated by Apache Airflow and containerised in a 6-service Docker stack. --- ## 🎯 Project Goal Kenya's property market has no centralised MLS (Multiple Listing Service). Buyers, analysts, and developers must manually cross-reference Pigiame, BuyRentKenya, Hassconsult, and several others — each with different structures, update frequencies, and price conventions (some in KSh, some monthly, some annual). This project eliminates that friction by building a repeatable ETL pipeline that answers: *what is the median sale price per square metre for a 3-bedroom house in Westlands, and how does Pigiame's asking price compare to BuyRentKenya's for the same category?* The two sources were chosen after live compatibility testing — Pigiame is fully server-rendered (fast, reliable with requests+BS4), while BuyRentKenya is a Livewire/Laravel SPA that returns an empty shell to plain HTTP requests and requires Playwright headless Chromium. All CSS selectors were verified against live HTML before any scraper code was written, which is why this project succeeds where two previous attempts failed. --- ## 🧬 System Architecture 1. **Ingestion Layer** — Two scrapers run as parallel Airflow tasks or sequentially via `run.py`. **Pigiame** (fully server-rendered HTML) uses **requests + BeautifulSoup4** against 5 category endpoints. **BuyRentKenya** (Liv …