YouanesAttia/egypt-tech-job-analyzer

An end-to-end Data Engineering pipeline that scrapes, processes, and visualizes the Egyptian job market. Features a Flask REST API and deep insights into skill demand and geographic trends. # Egypt Tech Job Market: Data Engineering & Analytics Pipeline A robust, end-to-end data pipeline designed to monitor and analyze the Software Development job market in Egypt. This project implements web scraping, data cleaning, statistical analysis, and a RESTful API to deliver actionable market intelligence. ## 🏗️ System Architecture The project is structured as a modular ETL pipeline: 1. **Extraction Layer:** Automated web scraping of the Wuzzuf job portal. 2. **Transformation Layer:** Cleaning, normalization, and feature engineering of unstructured job data. 3. **Analytical Layer:** Statistical processing and visualization of market trends. 4. **Service Layer:** A Flask-based REST API for data consumption. --- ## 🛠️ Technical Implementation ### 1. Data Acquisition (Scraper) - **Engine:** Built using `BeautifulSoup4` and `Requests`. - **Resiliency:** Implements an exponential backoff retry strategy to handle network instability and rate limiting. - **Logging:** Centralized logging system tracks scraping progress and records failed attempts in `data/log/scraper.log`. - **Validation:** Includes a runtime check for Cloudflare/JavaScript challenges and a fallback mechanism to local datasets. ### 2. Data Engineering (ETL) - **Normalization:** Standardizes job titles, company names, and skill tags into lowercase, stripped formats. - **Feature Engineering:** - **Geospatial Processing:** Splits raw location strings into distinct 'City', 'Governorate', and 'Country' features. - **Ordinal Encoding:** Maps categorical experience levels (e.g., 'Entry Level', 'Manager') to an integer-based 'Experience Index' for correlation analysis. - **Deduplication:** Removes redundant entries to maintain data integrity. ### 3. Analytics & Visualization - **Skill Correlation:** Computes co-occurrence matrices to identify common technology clusters. - **Geographical Analysis:** Quantifies market concentration across governorates. - **Visualization:** Utilizes `Seaborn` and ` …