Lanor-Jephthah1/nhis-payments-v2

Version 2 of fault-tolerant Selenium & Pandas scraper for NHIS Ghana payment records, with resumable checkpointing and clean CSV output. Handles pagination automatically and saves progress to a checkpoint file. Offers configurable timeouts, delays and deduplication # NHIS Payments Tracker v2 A robust, full-stack data pipeline and visualization dashboard designed to extract, clean, and visualize National Health Insurance Scheme (NHIS) payments across districts in Ghana. ## Live Demo Check out the live scraping and dashboard rendering in action: ## Data Integrity & The #REF! Issue During the extraction process, you may notice that some records in the generated CSV contain `#REF!` for the `District` and `Claim Month` fields. This is **not** a scraper bug. This corruption originates directly from the official NHIS source database. The NHIS administrators upload their payment data via Microsoft Excel spreadsheets, and broken cell references (`#REF!`) are being blindly imported into their backend and rendered directly on the public ASP.NET WebForms table. As photographic evidence, here is a screenshot of the official NHIS portal (Page 273, Size 20) actively serving these corrupted rows to the public: The pipeline implements automated sanitization on both the scraper and the backend API to dynamically detect and replace these `#REF!` and `########` string artifacts with "Unknown" to prevent them from crashing or corrupting downstream analytics dashboards. ## Architecture This project is built with resilience and scalability in mind, separated into three distinct microservices. ### 1. Selenium Scraper (`/scraper`) The extraction engine that pulls raw data from the official NHIS portal. - **Language:** Python - **Framework:** Selenium WebDriver with `webdriver-manager` - **Key Features:** - **Telerik API Fast-Forwarding:** Utilizes injected JavaScript to manipulate the ASP.NET `RadGrid` controls, allowing the scraper to instantly jump hundreds of pages (e.g., directly to page 1014) instead of manually clicking "Next" hundreds of times. - **Auto-Deduplication:** Aggressively runs a Pandas `drop_duplicates()` sweep on the entire CSV database every time a single page is saved. This mathematically guarantees 100% data integr …