AKWOKING/cameroon-job-miner

Automated web scraping and data mining system that tracks in-demand technical skills across four Cameroonian job portals. # 🇨🇲 Cameroon Tech Job Market Miner > Automated web mining & skill trend analysis for the Cameroonian technology sector > Academic project — Data Mining & Applications, 2025/2026 --- ## What it does Scrapes tech job listings from **4 Cameroonian job portals** every week, extracts in-demand skills using a bilingual (French/English) NLP pipeline, runs association rule mining and K-Means clustering, then presents everything in a **live public Streamlit dashboard**. | Portal | URL | |---|---| | Emploi.cm | emploi.cm | | Talent.cm | cm.talent.com | | Expertini.cm | cm.expertini.com | | WorkConnect | workconnectjob.com | --- ## Project phases | Phase | Weeks | Status | Built | |-------|-------|--------|-------| | **1 — Scraping** | 1–2 | ✅ | 4 scraper classes, SQLite storage, CSV export | | **2 — Processing** | 3–4 | ✅ | Cleaning, bilingual taxonomy, NLP skill extraction | | **3 — Mining & Dashboard** | 5–6 | ✅ | Apriori rules, K-Means, Streamlit dashboard | | **4 — Deploy & Document** | 7–8 | ✅ | Render.com config, GitHub Actions, full docs | --- ## Quick start ```bash # 1. Set up git clone github.com cd cameroon-job-miner python -m venv venv && venv\Scripts\activate # Windows pip install -r requirements.txt # 2. Scrape (Phase 1) python run_scrapers.py # 3. Clean + Mine (Phases 2 & 3) python run_pipeline.py # 4. Dashboard streamlit run app.py ``` --- ## Project structure ``` cameroon-job-miner/ ├── scrapers/ Phase 1 — web scraping │ ├── base_scraper.py httpx + BeautifulSoup base class │ ├── emploi_cm.py │ ├── talent_cm.py │ ├── expertini_cm.py │ └── workconnect.py auto-falls back to Selenium │ ├── pipeline/ Phases 2 & 3 — processing & mining │ ├── cleaner.py normalise, detect language, extract skills │ └── miner.py Apriori association rules + K-Means │ ├── data/ │ ├── …