Fikilesondach/sa-tech-job-market
Domain:
socioeconomicCreator:
Fik
Host:
Automated pipeline tracking the South African tech job market — Adzuna API → enrichment → Google Sheets → Looker Studio dashboard, refreshed weekly via GitHub Actions.
# SA Tech Job Market Analytics
A fully automated pipeline that tracks the South African technology job market across six role categories — pulling live listings from the Adzuna API every week, enriching them with salary, seniority, skills, and company classification, and surfacing insights in a Looker Studio dashboard.
Built as a portfolio project demonstrating end-to-end data engineering: API ingestion, structured enrichment, Google Sheets as a lightweight warehouse, and GitHub Actions for orchestration.
---
## Live dashboard
View on Looker Studio →
Refreshes automatically every Sunday at 06:00 SAST.
---
## What it tracks
| Category | Example roles |
|---|---|
| ML & AI | Data Scientist, ML Engineer, MLOps |
| Quant & Actuarial | Actuarial Analyst, Credit Risk, Quantitative Analyst |
| Cloud & DevOps | Cloud Engineer, Platform Engineer, SRE |
| Data | Data Analyst, Data Engineer, BI Developer |
| Software Engineering | Backend, Frontend, Full Stack |
| Cybersecurity | Security Analyst, SOC Analyst, Penetration Tester |
For each job the pipeline extracts:
- **Salary** — min/max/midpoint in annual ZAR (where disclosed, ~24% of SA listings)
- **Seniority** — Graduate / Junior / Mid / Senior / Lead
- **Skills** — matched against an 80-skill taxonomy across 9 domains
- **Company type** — Corporate / Startup / Consulting / Recruiting
- **Location** — province and city, normalised to standard SA metro names
---
## Architecture
```
Adzuna API
│
▼
src/ingestion/adzuna_client.py — 24 queries × up to 10 pages, 6-month rolling window
│
▼
src/cleaning/cleaner.py — flatten, normalise titles, parse salaries, detect remote
│
▼
src/enrichment/
├── category_classifier.py — ML & AI / Quant / Cloud / Data / SWE / Cyber
├── seniority_classifier.py — Graduate → Junior → Mid → Senior → Lead
├── skills_extractor.py — 80-skill keyword taxonomy (YAML-driven)
└── company_classifier.py — 90 known SA companies lookup table
│
▼
src/storage/sheets_writer.py …