Lanor-Jephthah1/nhis-payments-scraper

Fault-tolerant Selenium & Pandas scraper for NHIS Ghana payment records, with resumable checkpointing and clean CSV output. Handles pagination automatically and saves progress to a checkpoint file. Offers configurable timeouts, delays, and headless-mode operation for flexible deployment # NHIS Payments Scraper A robust, fault-tolerant **web scraper** built with Selenium and Pandas to extract payment records from the NHIS Ghana payments portal. Designed for resumable scraping, configurable delays, and clean data output. --- ## Table of Contents - Features - Demo - Prerequisites - Installation - Usage - Configuration - Directory Structure - Output - Customization - Contributing - License - Contact --- ## Features - **Complete Data Extraction** Scrapes all records: - Facility Name - District - Amount Paid - Claim Month - Payment Date - **Resumable & Incremental** Checkpoints progress to resume after interruptions. - **Configurable Delays & Timeouts** Prevents server overload; adapts to dynamic page loads. - **Data Validation & Cleaning** Filters out incomplete or blank rows before saving. - **Extensible & Maintainable** Modular design; easy to extend for additional columns or logic. --- ## Demo ```bash $ python scraper.py [INFO] Starting scrape at page 1 [INFO] Scraped page 1: 50 records ... [INFO] Completed. 1200 records saved to nhis_payments_all.csv ``` --- ## Prerequisites - **Python** ≥ 3.6 - **Google Chrome** - **ChromeDriver** matching your Chrome browser version - Python libraries: - `selenium` - `pandas` - `pytest` (for tests, optional) --- ## Installation 1. **Clone the repo** ```bash git clone github.com cd nhis-payments-scraper ``` 2. **Install dependencies** ```bash pip install -r requirements.txt ``` 3. **Verify ChromeDriver** Ensure `chromedriver` is in your `$PATH` or set `CHROMEDRIVER_PATH` environment variable. --- ## Usage Run with default settings: ```bash python scraper.py ``` View help for advanced options: ```bash python scraper.py --help ``` Command-line arguments: | Option | Description | Default | |-----------------------|----------------------------------------------|------------------ …