jrodriguez-ds/pump-it-up

Predicting water pump status in Tanzania — DrivenData competition | Score: 0.8264 # 💧 Pump it Up: Data Mining the Water Table Predicting the operational status of water pumps in Tanzania — DrivenData competition. ## Result | Metric | Value | |---|---| | **Official score (best)** | **0.8264** | | **Best submission** | S28 — Stacking RF + LightGBM (meta: Logistic Regression) | | **Total submissions** | 30 documented iterations | --- ## Problem Multiclass classification with three categories: - `functional` — working pump (54.3% of dataset) - `non functional` — broken pump (38.4%) - `functional needs repair` — working but needs attention (7.3%) The dataset contains ~59,400 records with approximately 40 variables covering geographic location, installation type, management structure, water quality and payment conditions. --- ## Process and key decisions ### 1. Exploration and data quality - Detection of **zeros masking missing values** in critical variables: `construction_year`, `longitude`, `latitude`, `gps_height`, `population` - Cardinality analysis and redundant variable identification (e.g. `payment` and `payment_type` with identical counts) - Single-value variable detection (`recorded_by`) eliminated before modeling - Interaction analysis between categorical variables using functionality rate variance as a proxy for predictive potential ### 2. Preprocessing - Merging semantically equivalent categories (`rope pump` → `handpump`) - Median imputation for numerical variables (robust to outliers) and `unknown` category for categoricals - Binary feature extraction from high-missing columns (`has_name`, `has_scheme`) - Removal of hierarchically redundant variables through cardinality ratio analysis ### 3. Feature Engineering - `pump_age`: difference between registration year and construction year - `qty_wpt`: `quantity_group × waterpoint_type` interaction — 3rd most important variable in final model - 6 additional interactions evaluated and discarded for not improving official score ### 4. Encoding - **Double-pass frequency encoding** for …