Brian-10-star/kenya-jobs-pipeline

# Kenya Jobs Pipeline 🇰🇪 An automated web scraping data pipeline that extracts real Kenyan tech job listings from BrighterMonday Kenya, transforms them into clean structured data, and loads them into a PostgreSQL database on a daily schedule. Built as part of a data engineering portfolio by Brian Mbugua Chira. ## What This Project Does Every day at 08:00 Nairobi time, this pipeline automatically: 1. **Extracts** job listings from the BrighterMonday Kenya IT & Software category. 2. **Transforms** the raw data — standardizing text, handling nulls, typing columns correctly. 3. **Loads** the cleaned data into PostgreSQL using an upsert strategy that prevents duplicates. This is a complete ETL (Extract, Transform, Load) pipeline built entirely in Python. ## Tech Stack | Tool | Purpose | |---|---| | Python 3.13 | Core programming language | | requests | Fetching webpage HTML | | BeautifulSoup4 | Parsing HTML structure | | Pandas | Data cleaning and transformation | | SQLAlchemy | Python-to-PostgreSQL connection | | psycopg2 | PostgreSQL database driver | | APScheduler | Daily pipeline scheduling | | python-dotenv | Secure credential management | | PostgreSQL 18 | Data storage | ## Project Structure kenya-jobs-pipeline/ ├── scraper.py # EXTRACT — fetches and parses job listings from BrighterMonday ├── transform.py # TRANSFORM — cleans, types, and standardizes raw data ├── load.py # LOAD — upserts cleaned data into PostgreSQL ├── pipeline.py # Orchestrator — runs all three ETL steps in sequence ├── scheduler.py # Runs pipeline.py automatically every day at 08:00 EAT ├── .env # Database credentials (not committed to GitHub) ├── requirements.txt # All Python dependencies with pinned versions └── README.md # Project documentation ## Pipeline Architecture BrighterMonday Kenya (IT & Software category) │ ▼ scraper.py — Fetches HTML, extracts job JSON from embedded Google Tag Manager data layer │ ▼ transform.py — Standar …