SU-CHF/harmonise-finscope

Harmonise Finscope Consumer South Africa surveys across years to create time series. # Harmonise FinScope This repository harmonises repeated cross-sectional FinScope Consumer South Africa surveys into a consistent time series of key indicators. The aim is to let researchers track financial inclusion, insurance take-up, livelihood indicators, and related behaviours through time without wrestling with changing survey wording. ## Project layout ``` mappings/ # easy-to-edit CSV lookup tables that drive harmonisation docs/ # narrative notes on survey waves and harmonisation decisions scripts/ # lightweight command line helpers (clean, harmonise, summarise) outputs/ # harmonised CSVs generated by the scripts utils.py # shared helpers (e.g. loading FinScope files) ``` Contributors can update the mappings directly in a spreadsheet editor. No need to touch application code unless the logic itself changes. ## Prerequisites 1. Python ≥3.10 with `pandas`, `numpy`, `pyreadstat`, `python-dotenv`. 2. Set the environment variable `DATA_PATH` (e.g. in a `.env` file) to the directory containing your FinScope survey extracts. The loaders expect files in `DATA_PATH/finscope/dta/FS_{year}.dta`. ## Typical workflow 1. **Point to the raw files** Ensure `DATA_PATH` references the OneDrive (or other shared location) that stores the FinScope extracts in `finscope/dta/FS_{year}.dta`. 2. **(Optional) Clean a single wave** Use the convenience script (or `make single`) to subset columns or export a smaller CSV/Parquet file: ``` make single # prompts for the year interactively # or: python scripts/clean_year.py 2019 --keep-columns H3a_13 i1_10 --output-format parquet ``` Outputs are written locally next to your project (default `outputs/finscope_{year}_clean.*`). 3. **Review or extend mappings** Open `mappings/harmonised_questions.csv` to adjust question codes, response values, or to add new indicators. Each row corresponds to a survey year and indicator, with human-readable labels and simple instructions (single column vs pre …