kudzaiprichard/loan-default-prediction

Distribution-shift-aware loan default prediction — adversarial validation revealed 91.5% train/test separation, guiding a LightGBM/CatBoost/XGBoost ensemble across 50+ Modal cloud experiments. Deep Learning IndabaX Zimbabwe 2026. Public LB 0.6840. # Loan Default Prediction — Systematic Distribution Shift Investigation **Deep Learning IndabaX Zimbabwe 2026 — AI for Financial Inclusion** **Public LB 0.6840 · Private LB 0.6878 · 50+ cloud experiments · ~$30 total compute cost** --- ## The Discovery That Shaped Everything Before training a single model, I ran **adversarial validation** — a technique where you train a binary classifier whose only job is to predict *"is this row from the training set or the test set?"* ```python # If a classifier can separate train from test, your OOF score is lying to you av_auc = train_adversarial_classifier(X_train, X_test) # → 0.9151 ``` **Train and test were 91.5% separable.** A near-perfect score. This meant the competition's training distribution and test distribution were from completely different time periods — the model would be trained on historical loan data and evaluated on future loan data with different borrower demographics, interest rate environments, and collateral mixes. Every OOF (out-of-fold) score in the competition was an optimistic lie. A model that scored 0.700 OOF might score 0.682 on the real leaderboard, and the gap would get *worse* the more you tuned. ```mermaid xychart-beta title "Features Driving Train/Test Separation (Adversarial Importance %)" x-axis ["dob_year", "rate_zscore", "collateral_type", "days_since_ref", "rate_residual", "approval_year", "approval_month"] y-axis "AV Feature Importance (%)" 0 --> 25 bar [21.92, 9.95, 9.71, 9.43, 9.07, 7.30, 5.20] ``` This single finding shaped every decision for the next 50 experiments. The problem was not "which model performs best?" — it was "how do we build a model that generalises across a temporal distribution shift?" --- ## Project Stats at a Glance | | | |---|---| | **Competition** | Deep Learning IndabaX Zimbabwe 2026 — AI for Financial Inclusion | | **Task** | Binary classification — loan default prediction (AUC metric) | | **Public Leaderboard AUC** | **0.6840** | | * …