Lanor-Jephthah1/nhis-payments-scraper
Domain:
healthcareRecord type:
software
Creator:
Lan
Host:
Fault-tolerant Selenium & Pandas scraper for NHIS Ghana payment records, with resumable checkpointing and clean CSV output. Handles pagination automatically and saves progress to a checkpoint file. Offers configurable timeouts, delays, and headless-mode operation for flexible deployment
# NHIS Payments Scraper
A robust, fault-tolerant **web scraper** built with Selenium and Pandas to extract payment records from the NHIS Ghana payments portal. Designed for resumable scraping, configurable delays, and clean data output.
---
## Table of Contents
- Features
- Demo
- Prerequisites
- Installation
- Usage
- Configuration
- Directory Structure
- Output
- Customization
- Contributing
- License
- Contact
---
## Features
- **Complete Data Extraction**
Scrapes all records:
- Facility Name
- District
- Amount Paid
- Claim Month
- Payment Date
- **Resumable & Incremental**
Checkpoints progress to resume after interruptions.
- **Configurable Delays & Timeouts**
Prevents server overload; adapts to dynamic page loads.
- **Data Validation & Cleaning**
Filters out incomplete or blank rows before saving.
- **Extensible & Maintainable**
Modular design; easy to extend for additional columns or logic.
---
## Demo
```bash
$ python scraper.py
[INFO] Starting scrape at page 1
[INFO] Scraped page 1: 50 records
...
[INFO] Completed. 1200 records saved to nhis_payments_all.csv
```
---
## Prerequisites
- **Python** ≥ 3.6
- **Google Chrome**
- **ChromeDriver** matching your Chrome browser version
- Python libraries:
- `selenium`
- `pandas`
- `pytest` (for tests, optional)
---
## Installation
1. **Clone the repo**
```bash
git clone github.com
cd nhis-payments-scraper
```
2. **Install dependencies**
```bash
pip install -r requirements.txt
```
3. **Verify ChromeDriver**
Ensure `chromedriver` is in your `$PATH` or set `CHROMEDRIVER_PATH` environment variable.
---
## Usage
Run with default settings:
```bash
python scraper.py
```
View help for advanced options:
```bash
python scraper.py --help
```
Command-line arguments:
| Option | Description | Default |
|-----------------------|----------------------------------------------|------------------ …