A hybrid AI factor framework for longitudinal analysis of multidimensional poverty status

Eliminating multidimensional poverty in Tanzania remains a significant challenge, hindering the country’s progress toward sustained economic growth. Over 47% of households experience multidimensional poverty. Achieving the eradication of this problem by 2030, as Stated by the Sustainable Development Goals 1, requires empirically testable strategies and shared national commitment. This study intends to predict multidimensional poverty along with identifying the key predictors. To achieve this, the study employs hybrid artificial intelligence (AI) techniques that integrates factor analysis, k-means clustering, H20AutoMl and its baseline models based on predictive modelling under unified hyper parameter optimization. The longitudinal survey data in 2014/15 and 2020/21 were analyzed in two setups. First, the data were analyzed independently, where training and testing was done for each wave, while in the second setup the first wave was treated as a training set and the second wave was treated as a testing set to uncover temporal effects. Results reveal that, analyzing the waves independently in the first setup, extreme gradient boosting (XGBoosting) outperforms other algorithms whereas including temporal variation in the second setup, Generalized Linear Model (GLM) outperformed other algorithms with Area Under the Curve (AUC) of 99.8%, F1-score for poor households of 98.3% and F1-score for non poor households of 96.9%, indicating its robustness to distributional shift across time. Furthermore, the key predictors reflect structural multidimensional poverty mechanisms, whereby household subsistence farming constrain income diversification, households with no formal education weaken human capital formation, loss of crop at the household level, occupation of the household, household residential area and household tenure induce livelihood shocks that reinforce multidimensional deprivation. The study suggests that the government and policy practitioners should adopt the application of GLM, which offers time varying effect benefits. Adopting GLM could help decision makers for programs and policy interventions, including promoting climate resilient farming and promotion of free education that is required to eradicate intergenerational multidimensional poverty.