ThubelihleNtabela/South-African-Job-Market-Intelligence-Platform

End-to-end Azure Data Engineering project for collecting and analyzing South African job market data using Python, SQL, and Streamlit. # South African Job Market Intelligence Platform End-to-end Azure Data Engineering and Analytics project for South African job market data. ## Project Overview This portfolio project addresses the need for reliable labour market intelligence in South Africa by ingesting job vacancy data from the Adzuna API. South African employers, recruiters, and labour market analysts need timely, accurate insights into hiring demand, salary expectations, and geographic distribution. The platform turns raw job listings into structured intelligence that supports actionable decision-making across hiring, workforce planning, and market research. The data pipeline is built using Python and Pandas to orchestrate ingestion, transformation, and storage. Job listings are collected from the Adzuna API, saved as raw JSON for traceability, then processed through an ETL workflow that cleans records, removes duplicates, and handles missing values. Processed data is exported to a clean CSV file and uploaded to Azure Blob Storage before being loaded into an Azure SQL Database for analytics and dashboarding. The Streamlit dashboard provides a recruiter-friendly analytics layer with interactive charts and filters powered by Plotly. Users can explore total jobs, average salary, top companies, job categories, location trends, and skill demand. This end-to-end solution demonstrates modern Azure data engineering practices while making South African job market insights accessible and easy to interpret. The current version collects and processes approximately 775 South African job listings using multi-page Adzuna API ingestion. ## Project Status Completed MVP with cloud pipeline and dashboard ## Architecture ```mermaid flowchart LR A[Adzuna API] --> B[Python ETL Pipeline] B --> C[Azure Blob Storage] C --> D[Azure SQL Database] D --> E[Streamlit Dashboard] ``` ## 📸 Project Screenshots The following screenshots demonstrate the end-to-end Azure Data Engineering pipeline and the resulting analyt …