AKWOKING/cameroon-job-market-analysis

Automated web scraping and data mining system that tracks in-demand technical skills across four Cameroonian job portals. # Cameroon Tech Job Market Miner Automated web scraping and data mining system that tracks in-demand technical skills across four Cameroonian job portals. ## Portals covered | Portal | URL | |---|---| | Emploi.cm | emploi.cm | | Talent.cm | cm.talent.com | | Expertini.cm | cm.expertini.com | | WorkConnect | workconnectjob.com | --- ## Setup (Windows + VS Code) ### 1. Clone / open the project ``` File → Open Folder → select cameroon-job-miner/ ``` ### 2. Create a virtual environment Open the VS Code terminal (`` Ctrl+` ``) and run: ```bash python -m venv venv venv\Scripts\activate ``` You should see `(venv)` at the start of your prompt. ### 3. Install dependencies ```bash pip install -r requirements.txt ``` ### 4. Run all scrapers ```bash python run_scrapers.py ``` ### 5. Run a single scraper (for testing) ```bash python run_scrapers.py --portal emploi_cm python run_scrapers.py --portal talent_cm python run_scrapers.py --portal expertini_cm python run_scrapers.py --portal workconnect ``` --- ## Output | File | Description | |---|---| | `data/jobs.db` | SQLite database — all job records, deduplicated | | `data/exports/jobs_YYYYMMDD_HHMMSS.csv` | Timestamped CSV export after each run | | `scraper.log` | Full request log for debugging | ### Database schema (`jobs` table) | Column | Type | Notes | |---|---|---| | `id` | INTEGER | Auto-increment primary key | | `hash` | TEXT | Deduplication key (title+company+source) | | `title` | TEXT | Job title | | `company` | TEXT | Employer name | | `city` | TEXT | Location | | `experience` | TEXT | Experience level | | `skills_raw` | TEXT | Raw skills text from the page | | `description` | TEXT | Full job description | | `url` | TEXT | Direct link to listing | | `date_posted` | TEXT | Posting date (raw string) | | `source` | TEXT | Portal key (e.g. `emploi_cm`) | | `scraped_at` | TEXT | ISO timestamp of the scraping run | --- ## Configuration (`config/settings.py`) | Setting | …