DeleLinus/HFR-Data-Warehousing

End-to-end data engineering processes for the NIGERIA Health Facility Registry (HFR). The project leveraged Selenium, Pandas, PySpark, PostgreSQL and Airflow [//]: # (I must include tools section and add tools images, check proper README design tutorial) [//]: # (images and style from simpleicons.org) ## Table of Contents 1. Project Description 2. About Nigeria HFR 3. Project Development 4. Environment Setup 5. Run Programs 6. References # NIGERIA Health Facility Registry (HFR) Data Warehousing (*in progress*) This is an end-to-end data engineering project that entails the design and implementation of a data warehouse for the Nigeria Health Facility Registry which is itself a program under the Nigeria Ministry of Health. The goal of this project is to develop a complete reporting solution to help the Ministry monitor and manage the contry's Master Health Facility List (MFL). This project development emcompasses: 1. The design of a **data ingestion pipeline architecture** showing the different tools and framework I would be proposing to the organization in achieving the result. 2. The design of the **ERD(Entity Relationship Diagram)** of the system and schema of my final data mart that would be used for reporting. 3. **The implementation of Data Extraction, Transformation and Loading (ETL)** 4. The **Automation and Scheduling** of the processes # About Nigeria HFR The Nigeria Health Facility Registry (HFR) was developed in 2017 as part of effort to dynamically manage the MFL in the country. The MFL "is a complete listing of health facilities in a country (both public and private) and is comprised of a set of identification items for each facility (signature domain) and basic information on the service capacity of each facility (service domain)" . The Federal Ministry of Health had previously identified the need for an information system to manage the MFL in light of different shortcomings encountered in maintaining an up-to-date paper based MFL. The benefits of the HFR are numerous including serving as the hub for connecting different information systems thereby enabling integration and …