Moazabdelaleem/Egypt-Job-Market-Pipeline
Domaine:
socioeconomicnatural language processingType de record:
project
Créateur:
Moa
Hôte:
A data pipeline represents and analyze the job market in egypt showing its trends and needs
# 📊 Egypt Job Market Intelligence Pipeline
> An automated, end-to-end data engineering pipeline that harvests job postings from **multiple Egyptian job posting websites**, processes them through a **Medallion Data Warehouse**, and powers a live **Power BI dashboard** with real-time market intelligence.
---
## 🧰 Tech Stack
| Layer | Technology |
|---|---|
| **Web Scraping** | Scrapy + Playwright (headless browser) |
| **NLP & Skill Extraction** | Rule-based Regex NLP (60+ skills) |
| **Cloud Database** | Supabase (PostgreSQL) |
| **Data Warehouse** | Bronze → Silver → Gold (Medallion Architecture) |
| **Normalization** | Custom Python NLP (`normalize.py`) |
| **Visualization** | Power BI (Star Schema, DAX measures) |
| **Automation** | GitHub Actions (scheduled every 2 days) |
---
## 🏗️ Architecture & Data Flow
The pipeline follows a **Medallion Architecture**, moving data from raw extraction through progressive refinement into analytics-ready layers.
```
[Job Postings Websites]
│ Scrapy + Playwright crawl
▼
┌─────────────────────────────────────────┐
│ BRONZE LAYER — job_postings │ Raw scraped data
│ Auto-created by spider on first run │ job_hash, date_first_seen,
│ UPSERT via job_hash deduplication │ date_last_seen, status
└─────────────────────────────────────────┘
│ normalize.py (auto-triggered after crawl)
▼
┌─────────────────────────────────────────┐
│ SILVER LAYER — job_postings_clean │ Normalized & enriched
│ NLP classifies job family, seniority │ job_family, seniority_level,
│ Extracts city/governorate, salary, │ city, governorate,
│ education, and inferred skills │ salary_min/max, education
└─────────────────────────────────────────┘
│ Analytical SQL Views
▼
┌─────────────────────────────────────────┐
│ GOLD LAYER — Star Schema Views │ Power BI consumption
│ Fact table + 8 dimension/bridge views │ DAX measures, KPIs,
│ Live connection from Power BI │ interactive dashboard …