Momahmoses/nigeria-job-market-intelligence
Domain:
socioeconomicRecord type:
software
Creator:
Mom
Host:
||Real-time tech job market intelligence for West Africa covering scraping, streaming, storage and analysis of job listings.
# Nigeria Job Market Intelligence Platform
A real-time job market data platform for West Africa, scraping, streaming, storing, and analyzing tech job listings across Nigeria and the region.
## Problem
Job seekers, HR consultants, and bootcamps lack real-time visibility into which skills are in demand, what salaries look like, and which companies are actively hiring.
## Architecture
```
Scrapers → Kafka Stream → S3 Data Lake → PostgreSQL Warehouse → Analysis/Dashboard
```
## Features
- **Web scraper**: Pulls listings from Jobberman, LinkedIn, Indeed Nigeria
- **Kafka pipeline**: Streams new listings in real-time
- **PostgreSQL warehouse**: Full-text search, skill indexing, salary queries
- **Analysis engine**: Salary benchmarks, skill demand trends, company rankings
- **Charts**: Auto-generated market intelligence visualizations
## Quick Start
```bash
pip install -r requirements.txt
# Generate 5,000 synthetic job listings (no scraping needed)
python src/pipeline/generate_sample_data.py
# Run analysis and generate charts
python src/pipeline/analysis.py
# Initialize warehouse and load data
python src/warehouse/db.py
# Run live scraper (respects rate limits)
python src/scraper/scraper.py
# Stream to Kafka (requires running broker)
python src/pipeline/kafka_producer.py
```
## Sample Output
| Role | Median Salary (₦/month) | Jobs |
|------|------------------------|------|
| Cloud Architect | ₦875,000 | 312 |
| AI Research Engineer | ₦820,000 | 198 |
| ML Engineer | ₦750,000 | 445 |
| Data Scientist | ₦620,000 | 587 |
| Data Engineer | ₦610,000 | 401 |
## Key Insights (from synthetic data)
- Python is the #1 skill across all roles (72% of listings)
- ML/AI roles pay 40% more than average tech roles
- Remote roles pay 25% more than on-site equivalents
- Lagos has 60% of all tech job listings
## Docker (Kafka + PostgreSQL)
```yaml
# docker-compose.yml included — run:
docker-compose up -d
```
## Real Impact
- Career coaches use it to advise students on skill g …