Fikilesondach/sa-tech-job-market

Automated pipeline tracking the South African tech job market — Adzuna API → enrichment → Google Sheets → Looker Studio dashboard, refreshed weekly via GitHub Actions. # SA Tech Job Market Analytics A fully automated pipeline that tracks the South African technology job market across six role categories — pulling live listings from the Adzuna API every week, enriching them with salary, seniority, skills, and company classification, and surfacing insights in a Looker Studio dashboard. Built as a portfolio project demonstrating end-to-end data engineering: API ingestion, structured enrichment, Google Sheets as a lightweight warehouse, and GitHub Actions for orchestration. --- ## Live dashboard View on Looker Studio → Refreshes automatically every Sunday at 06:00 SAST. --- ## What it tracks | Category | Example roles | |---|---| | ML & AI | Data Scientist, ML Engineer, MLOps | | Quant & Actuarial | Actuarial Analyst, Credit Risk, Quantitative Analyst | | Cloud & DevOps | Cloud Engineer, Platform Engineer, SRE | | Data | Data Analyst, Data Engineer, BI Developer | | Software Engineering | Backend, Frontend, Full Stack | | Cybersecurity | Security Analyst, SOC Analyst, Penetration Tester | For each job the pipeline extracts: - **Salary** — min/max/midpoint in annual ZAR (where disclosed, ~24% of SA listings) - **Seniority** — Graduate / Junior / Mid / Senior / Lead - **Skills** — matched against an 80-skill taxonomy across 9 domains - **Company type** — Corporate / Startup / Consulting / Recruiting - **Location** — province and city, normalised to standard SA metro names --- ## Architecture ``` Adzuna API │ ▼ src/ingestion/adzuna_client.py — 24 queries × up to 10 pages, 6-month rolling window │ ▼ src/cleaning/cleaner.py — flatten, normalise titles, parse salaries, detect remote │ ▼ src/enrichment/ ├── category_classifier.py — ML & AI / Quant / Cloud / Data / SWE / Cyber ├── seniority_classifier.py — Graduate → Junior → Mid → Senior → Lead ├── skills_extractor.py — 80-skill keyword taxonomy (YAML-driven) └── company_classifier.py — 90 known SA companies lookup table │ ▼ src/storage/sheets_writer.py …