harryoghonyon/MarketPulse

A Python-based marketplace data pipeline for scraping, cleaning, validating, deduplicating, and storing structured Jiji Nigeria listing data. # MarketPulse NG MarketPulse NG is a small Python data ingestion pipeline for collecting publicly visible marketplace listing data from Jiji Nigeria, converting semi-structured web data into a clean schema, validating records, deduplicating them, storing them in SQLite, and exporting analysis-ready CSV data. This is an independent portfolio project. It is not affiliated with, endorsed by, or sponsored by Jiji. ## Architecture ```text Jiji | v Extraction | v Parsing | v Cleaning / Transform | v Validation | v Deduplication | v SQLite / \ v v CSV Analysis ``` ## Features - HTTP extraction with timeouts, retries, request headers, and configurable delay. - HTML parsing separated from fetching. - Deterministic price, location, category, and whitespace normalization. - Explicit listing validation with invalid-record reporting. - SQLite storage with unique constraints for deduplication. - CSV export to `data/processed/listings.csv`. - CLI with configurable location, category, pages, query, delay, database path, and CSV path. - Fixture-based tests that do not depend on Jiji being online. - Simple SQL-based analysis script. - Dockerfile and GitHub Actions test workflow. ## Tech Stack - Python - Requests - BeautifulSoup - SQLite - Pytest - Docker - GitHub Actions ## Installation ```bash git clone cd MarketPulse python -m venv .venv source .venv/bin/activate python -m pip install --upgrade pip python -m pip install -e ".[dev]" ``` ## Usage Run against live public listing pages: ```bash python -m marketpulse --location lagos --category mobile-phones --pages 2 --delay 2 ``` Run against the local fixture, useful for development and demos: ```bash python -m marketpulse --fixture tests/fixtures/jiji_search_sample.html ``` Example runtime output: ```text MarketPulse NG pipeline completed Listings extracted: 3 Valid listings: 2 Invalid listings: 1 Duplicates: 0 Stored records: 2 Exported records: 2 ``` ## Data Model Each valid listing is …