ibagur/sudan-wash-survey-unsupervised

Unsupervised clustering analysis of household WASH survey data from Tawila IDP camp, Sudan (Feb 2026) — PCA + K-Means vulnerability segmentation with TabPFN supervised classifier and SHAP interpretability # Tawila WASH Survey — Clustering & Classification Analysis End-to-end analysis of household-level WASH (Water, Sanitation, and Hygiene) survey data from Tawila IDP camp, Sudan (February 2026). The project combines an **unsupervised clustering pipeline** to identify distinct household vulnerability groups with a **supervised TabPFN classifier** that can rapidly assign new households to those groups. --- ## Overview - **369 households** surveyed across four camp sections (Camps A–D) - **234 survey variables** covering five WASH domains - **K=2 primary result** (silhouette 0.595), **K=3 refinement** (silhouette 0.554) - **86.2% agreement** between K-Means and hierarchical clustering (validation) - **Strong geographic signal**: Chi-square p = 2.3×10⁻²⁴ — clusters align with camp sections - **TabPFN classifier** trained on cluster labels achieves high predictive performance with SHAP-based feature importance --- ## Scripts | Script | Role | |---|---| | `tawila_wash_clustering_pipeline.py` | Stage 1–10 unsupervised pipeline: feature engineering → PCA → K-Means → profiling | | `tabpfn_cluster_classifier.py` | Follow-up supervised classifier: predicts cluster membership for new households | --- ## Quick Start ### Requirements Python 3.11+ ```bash # Unsupervised pipeline pip install pandas numpy scikit-learn scipy matplotlib seaborn prince openpyxl # TabPFN classifier (additional) pip install tabpfn torch "tabpfn-extensions[interpretability]" ``` ### Run ```bash # Step 1 — run the clustering pipeline (generates cluster assignments) python tawila_wash_clustering_pipeline.py # Step 2 — train and evaluate the cluster classifier python tabpfn_cluster_classifier.py ``` The clustering pipeline reads from `data/20260210_wash_survey_hh_container_level_PROCESSED.xlsx` and writes all outputs to `output/`. The classifier reads `data/households_with_cluster.csv` (produced by the pipeline) and appends its outputs to `output/`. --- ## Input Data | Sheet | Rows | Colum …