GigiMW/egypt-realestate-pipeline
Domain:
socioeconomicRecord type:
software
Creator:
Gig
Host:
# Egypt Real Estate Pipeline
An end-to-end data engineering pipeline that scrapes live apartment listings
from Dubizzle Egypt, transforms them with dbt, and produces Cairo market analytics.
## Architecture
Dubizzle Egypt
│
▼
Python Scraper (BeautifulSoup)
│
▼
DuckDB ──► raw_listings
│
▼
dbt Core
├──► stg_listings (cleaned prices, areas, timestamps)
├──► mart_listings (price buckets, price per sqm)
├──► avg_price_by_location (aggregated by Cairo district)
└──► price_distribution (listing count by price range)
│
▼
Apache Airflow DAG (daily @ 8am)
## Stack
| Layer | Tool | Purpose |
|-------|------|---------|
| Ingestion | Python, BeautifulSoup | Scrape live Dubizzle listings |
| Storage | DuckDB | Local analytical database |
| Transformation | dbt Core 1.11 | Staging + mart models |
| Orchestration | Apache Airflow 2.9 | Daily scheduling + retries |
| Containerisation | Docker Compose | One-command local setup |
## Data Models
### Staging
- `stg_listings` — raw listings cleaned: prices parsed to EGP float,
areas extracted to sqm integers, price-per-sqm calculated
### Marts
- `mart_listings` — all listings enriched with price bucket classification
- `avg_price_by_location` — average price and price/sqm by Cairo district
- `price_distribution` — listing count and avg price per price range bucket
## dbt Lineage
## Key Metrics Produced
| Metric | Description |
|--------|-------------|
| Price per sqm | EGP per square meter per listing |
| Price bucket | under_3M / 3M_to_5M / 5M_to_10M / 10M_plus |
| Avg price by district | Aggregated by Cairo location |
| Distribution | Count and avg price per price range |
## Sample Data (Latest Run)
| Price Bucket | Listings | Avg Price EGP | Avg EGP/sqm |
|-------------|----------|---------------|-------------|
| 10M+ | 7 | 17,394,325 | 104,729 |
| 5M–10M | 6 | 8,171,667 | 55,649 |
| 3M–5M | 5 | 4,010,000 | 30,814 |
## How to Run Locally
### Option 1 — Manual (Development)
```bash
# Clone and setup
git …