CodeForAfrica-SCRAPERS/healthtools_ke

[morph] Scrapers for the HealthTools Kenya data. # HealthTools Kenya Scraper This is a suite of scrapers that retrieve actionable information for citizens to use. All the data scraped by this is accessible through our HealthTools API. They retrieve data from the following sites: - Doctors: medicalboard.co.ke - Foreign Doctors: medicalboard.co.ke - Health Facilities: kmhfl.health.go.ke - NHIF accredited inpatient facilities: nhif.or.ke - NHIF accredited outpatient facilities: nhif.or.ke - NHIF accredited outpatient facilities for civil servants: nhif.or.ke They currently run on morph.io but you are able to set it up on your own server. ## Getting Started These instructions will get you a copy of the project up and running on your local machine for development and testing purposes. ### How the Scrapers Work To get the data we follow a couple of steps: **1. Scrape website:** This is done in most cases using Beautiful Soup. **2. Elastic Index update:** Replace data on elasticsearch with the new one. We only delete the documents after succesful completion of the scraping and not before. In the doctors' case, because we pull together foreign and local doctors, we won't update elasticsearch until both have been scraped succesfully. **3. Archive data:** We archive the data in a `latest.json` file so that the url doesn't have to change to get the latest version in a "dump" format. A date-stamped archive is also stored as we later intend to do some analysis on the changes over time. Should the scraper fail at any of these points, we log the error, and if set up, a Slack notification is sent. --- # Development Clone repo and install the requirements this way: ```sh $ git clone git@github.com:CodeForAfrica-SCRAPERS/healthtools_ke.git $ cd healthtools_ke $ mkvirtualenv healthtools-k …