Moazabdelaleem/Egypt-Job-Market-Pipeline

A data pipeline represents and analyze the job market in egypt showing its trends and needs # 📊 Egypt Job Market Intelligence Pipeline > An automated, end-to-end data engineering pipeline that harvests job postings from **multiple Egyptian job posting websites**, processes them through a **Medallion Data Warehouse**, and powers a live **Power BI dashboard** with real-time market intelligence. --- ## 🧰 Tech Stack | Layer | Technology | |---|---| | **Web Scraping** | Scrapy + Playwright (headless browser) | | **NLP & Skill Extraction** | Rule-based Regex NLP (60+ skills) | | **Cloud Database** | Supabase (PostgreSQL) | | **Data Warehouse** | Bronze → Silver → Gold (Medallion Architecture) | | **Normalization** | Custom Python NLP (`normalize.py`) | | **Visualization** | Power BI (Star Schema, DAX measures) | | **Automation** | GitHub Actions (scheduled every 2 days) | --- ## 🏗️ Architecture & Data Flow The pipeline follows a **Medallion Architecture**, moving data from raw extraction through progressive refinement into analytics-ready layers. ``` [Job Postings Websites] │ Scrapy + Playwright crawl ▼ ┌─────────────────────────────────────────┐ │ BRONZE LAYER — job_postings │ Raw scraped data │ Auto-created by spider on first run │ job_hash, date_first_seen, │ UPSERT via job_hash deduplication │ date_last_seen, status └─────────────────────────────────────────┘ │ normalize.py (auto-triggered after crawl) ▼ ┌─────────────────────────────────────────┐ │ SILVER LAYER — job_postings_clean │ Normalized & enriched │ NLP classifies job family, seniority │ job_family, seniority_level, │ Extracts city/governorate, salary, │ city, governorate, │ education, and inferred skills │ salary_min/max, education └─────────────────────────────────────────┘ │ Analytical SQL Views ▼ ┌─────────────────────────────────────────┐ │ GOLD LAYER — Star Schema Views │ Power BI consumption │ Fact table + 8 dimension/bridge views │ DAX measures, KPIs, │ Live connection from Power BI │ interactive dashboard …