declerke/Africa-Skills-NLP

Production-grade NLP pipeline scraping 357 tech jobs from 6 African job boards. spaCy skill extraction, Airflow 3.0, dbt 1.9, Streamlit dashboard # 🌍 Africa Skills Intelligence: Tech Job Market Analytics Pipeline **Africa Skills Intelligence** is a production-grade data engineering pipeline that scrapes live tech job postings from six African job boards, applies a multi-stage NLP processing chain to extract skill mentions, classify job roles, and parse salary figures, then transforms the enriched data through a dbt layer and surfaces the results across a five-tab Streamlit analytics dashboard. It implements a **scrape → NLP → transform → visualise** architecture — raw HTML and JSON from job listing pages feed a PostgreSQL warehouse, a spaCy EntityRuler extracts structured skills, keyword-based role classification assigns every posting to one of ten standardised tech roles, and dbt aggregates the data into mart tables powering real-time insights on skill demand, employer distribution, salary ranges, and hiring locations across Kenya and East Africa. | Metric | Value | |--------|-------| | Job postings scraped | **357** (across 6 sources) | | Unique employers | **94** | | Skills identified | **19** distinct skills across 6 categories | | Job roles classified | **357** (10 role categories) | | Postings with date | **81** (CorporateStaffing · RemoteOK) | | Postings with salary | **6** (RemoteOK USD ranges) | | Airflow DAG tasks | **6** | | dbt models | **8** (2 staging · 1 intermediate · 5 marts) | | dbt tests | **59/59 PASS** | | Dashboard tabs | **5** | | API cost | **$0** — open sources + local stack | --- ## 🎯 Project Goal Tech hiring in Kenya and across East Africa is fragmented across dozens of job boards, each with a different format, different listing structure, and no unified view of which skills employers are actually requesting. A data engineer trying to understand what Python frameworks are in demand in Nairobi, which companies are the most active tech hirers on the continent, or how seniority levels map to salary ranges has no single source of truth. Africa Skills Intelligence automates that wo …