danaelshrbiny10/Market-Driven-Data-Skills-Analysis

Discover the most in-demand Data Analytics skills and tools in Egypt using real LinkedIn jobs and GitHub projects. # Market Driven Data Skills Analysis This project collects and analyzes **Data Analytics skills and tools** from two sources: 1. **LinkedIn Egypt job postings** – to identify popular tools and skills required by employers. 2. **GitHub repositories** – to extract skills and tools used by developers in real-world projects. The goal is to create a **comprehensive dataset** of tools, skills, and experience levels (Junior/Mid/Senior) to guide learners and professionals entering the data field in Egypt. ## ⚙️ Setup & Installation 1. **Clone the repository** ```bash git clone github.com Market-Driven-Data-Skills-Analysis.git ``` 2. Create a virtual environment ```bash python3 -m venv env source env/bin/activate # Linux/macOS env\Scripts\activate # Windows ``` ## 📦 Install package in editable mode Editable mode (recommended for development) ```bash pip install -e . ``` - Allows changes in the source code to take effect immediately. - Installs all dependencies (pandas, requests, python-dotenv, serpapi). ## 🚀 Directly from GitHub ```bash pip install git+github.com ``` - Installs the latest stable version of the package from GitHub. 3. Install dependencies ```bash pip install -r src/requirements.txt ``` 4. **Create a `.env` file** in the project root: ```bash SERPAPI_KEY=YOUR_SERPAPI_KEY GITHUB_TOKEN=YOUR_GITHUB_PERSONAL_ACCESS_TOKEN ``` ## 🚀 Running the Scrapers 1. **LinkedIn Egypt Jobs Scraper** ```bash python3 src/scraping/linkedin_scraper.py ``` - Fetches Data Analytics job postings from LinkedIn Egypt via SerpAPI Google Jobs. - Extracts: - Job title, company, location, description - Level: Junior / Mid / Senior - Job URL (Google Search link pointing to LinkedIn) - Saves CSV: src/data/raw/linkedin_egypt_jobs.cs 2. **GitHub Repositories Scraper** ```bash python3 src/scraping/github_scraper.py ``` - Fetches repositories related to data analytics from GitHub. - …