jooshawqi/egypt-net-migration-analysis

Data science project analyzing Egypt's net migration (1960-2025) using time-series feature engineering, ML models, and ARIMA forecasting # Egypt Net Migration Analysis A complete data science pipeline analyzing Egypt's net migration time-series (1960–2025), combining exploratory data analysis, statistical testing, and predictive modeling with both machine learning and classical time-series methods. **Data Source**: World Bank — World Development Indicators (Indicator: `SM.POP.NETM`) ## Overview This project analyzes 66 years of net migration data for Egypt to uncover historical patterns and evaluate how well different modeling approaches can forecast future migration trends. ## Key Findings - Dataset spans **1960–2025** with zero missing values - The series is statistically **stationary** (ADF test: p = 0.0001) - **Ridge Regression** achieved the best generalization on the test set (R² = 0.997, MAE ≈ 5,168) - Two major migration events stand out: peak inflow in **1990** (+406,994 — mass return of Egyptian workers from Kuwait after the Iraqi invasion) and peak outflow in **2010** (–194,100) - The most predictive features were **lag1** (prior-year value) and the **3-year rolling mean** ## Pipeline Stages 1. **Data Loading & Cleaning** — Load raw World Bank CSV, extract Egypt's time series, handle missing values 2. **Feature Engineering** — Lag features (1, 2, 3, 5 years), rolling means (3/5/10 years), differencing, year-on-year % change, decade bins 3. **Exploratory Data Analysis** — Time-series plot with rolling averages, decade boxplots, distribution histogram, ACF, outlier detection (IQR), YoY change 4. **Stationarity Testing** — Augmented Dickey-Fuller (ADF) test 5. **Predictive Modeling** — Time-series-aware train/test split, four ML models plus an ARIMA baseline 6. **Model Evaluation** — MAE, RMSE, R², cross-validated MAE (TimeSeriesSplit), feature importance, residual analysis ## Models Compared | Model | MAE | RMSE | R² | |---|---|---|---| | Linear Regression | 0 | 0 | 1.000* | | **Ridge Regression** | **5,168** | **6,669** | **0.997** | | Random Forest | 65,418 | 84,126 | 0.516 | | G …