mohamed-mahmoud-de/Egypt-Job-Pipeline

Data pipeline that scrapes Egyptian tech job listings and stores them in Postgres # Egypt Jobs Pipeline > A data pipeline that scrapes Egyptian tech job listings from Wuzzuf.net, stores them in Postgres, and sends real-time alerts when new jobs appear. Built as a portfolio project to learn data engineering fundamentals. --- ## Overview This pipeline automates the collection of tech job listings in Egypt from Wuzzuf. It has evolved through three versions: - **V1** scraped jobs on demand and loaded them into Postgres. - **V2** added Airflow orchestration, running every 2 hours with retries, Discord alerts, backfill, and a Streamlit dashboard. - **V3** added a real-time alert layer — a polling loop that checks Wuzzuf every 3 minutes, deduplicates with Redis, filters by keywords, and sends Discord notifications within minutes of a new listing appearing. ## Architecture ### V2 — Batch Pipeline (Airflow) ``` ┌─────────────┐ ┌───────────────────────────────────────────────┐ │ Airflow │ │ DAG (every 2h) │ │ Scheduler │───>│ Extract ──> Transform ──> Load ──> Notify │ └─────────────┘ └──────┬────────────────────────┬────────┬──────┘ │ │ │ Wuzzuf.net PostgreSQL Discord (all pages) (permanent) (success/fail) ``` ### V3 — Real-Time Alerts (Polling Loop) ``` ┌──────────────────────────────────┐ │ poller.py (while True) │ │ runs every 3 minutes │ └───────────────┬──────────────────┘ │ scraper.py │ Wuzzuf.net (page 1 only) │ ┌─────▼──────┐ │ For each │ │ job found │ └─────┬──────┘ │ ┌──────────▼──────────┐ │ redis_client.py │ │ is_seen(job_id)? │ │ (Redis Set lookup) │ └──────┬───────┬──────┘ │ │ Yes │ │ No Skip │ ▼ │ filter.py │ matches keywords? │ │ │ Yes │ No │ │ │ mark_seen + skip │ ▼ │ │ notifier.py │ send Discord alert │ │ │ ▼ │ mark_seen(job_id) │ ┌──────▼──────────────┐ │ On crash: │ │ send_error_alert() │ │ wait 3 min, retry │ …