neotree/neotree-cleaning-pipeline

R pipelines for cleaning, validating and subsampling neonatal clinical data from Neotree deployments in Malawi and Zimbabwe # Neotree Cleaning Pipeline **Version 1.1.0** — see CHANGELOG.md. Each run also stamps the version into its own log, so a cleaned dataset can be traced back to the code that produced it. R pipeline for cleaning and validating neonatal clinical data collected through Neotree deployments in **Malawi** and **Zimbabwe**. It takes raw data exported from the Neotree PostgreSQL database (directly or via Metabase) and produces a faithful, typed, dictionary-conformant cleaned dataset. ``` raw Neotree export → cleaning_pipeline_R → cleaned dataset → neotree-sample-maker (CSV / Metabase) (validate & type) (output/ CSV + RDS) (separate repository) ``` Downstream work — joining admissions to discharges, building master datasets and cutting analysis-ready subsamples — lives in a separate repository: **neotree-sample-maker**. The two are loosely coupled: this pipeline writes cleaned datasets to `output/`, and the sample maker reads whatever it is given. Neither requires the other to be present to run. --- ## What it does Sequential, numbered modules, each self-documented and runnable in order: | Stage | Modules | What happens | |---|---|---| | Preparation | `00_build_dictionary`, `00_setup` | Build the data dictionaries the pipeline validates against | | De-identification | `00a_pii_detection_removal` | PII detected and removed **first**, ahead of any cleaning | | Structural repair | `00b`–`03` | Harmonise column names, correct frame shifts, merge duplicate columns | | Value cleaning | `04`–`08` | Reduce values to dictionary canonical codes, forward-fill, drop label and autopopulated columns | | Typing and validation | `09`–`14a` | Assign data types, remove duplicate rows, validate numeric, boolean, categorical and datetime fields | | Output | `15`, `16` | Final merge, derived canonical columns, NA-reason coding | Two design decisions worth knowing before you use the output: **Categorical harmonisation is decision-free.** Values are reduced …