A Multimodal Approach to Knowledge Tracing for Personalized Learning based on Cognitive and Affective Data in Resource-Constrained Environments

Abstract Personalized learning remains a critical challenge in education, particularly in resource-constrained environments such as Zambia, where instructional practices often follow a one-size-fits-all approach that overlooks differences in learners’ cognitive processes, behaviours, and emotional states. In STEAM education, this limitation contributes to learner disengagement and poor academic performance. Although knowledge tracing (KT) techniques provide data-driven methods for modeling student learning, most existing approaches focus primarily on cognitive outcomes and fail to capture the multimodal nature of the learning process. This study proposes a multimodal approach to knowledge tracing for personalized learning that integrates cognitive and affective data to provide a more comprehensive representation of learner behaviour. The proposed framework incorporates five modalities: knowledge mastery estimates, programming interaction features, textual help-seeking patterns, affective states, and behavioural indicators. These heterogeneous data sources are combined using a context-aware deep learning architecture with hierarchical fusion and adaptive gating, enabling dynamic weighting of modalities and robustness to missing data-an important requirement in resource-constrained settings. The model was evaluated on three benchmark datasets (ASSISTments 2015, CSEDM 2019, and XES3G5M) using five-fold cross-validation on 1,065 student interaction sequences. The results demonstrate strong predictive performance, achieving a mean AUC-ROC of 0.9933 ± 0.0006, substantially outperforming traditional Bayesian Knowledge Tracing and standard Deep Knowledge Tracing models. Analysis of modality contributions shows that affective states (43.7%) and behavioural indicators (26.3%) are the most influential predictors of learning outcomes, providing empirical evidence of the importance of non-cognitive factors in personalized learning. These findings highlight the value of multimodal knowledge tracing in enhancing personalization and improving learner modelling, particularly in environments with limited educational resources. The proposed approach offers a scalable and context-aware solution for intelligent tutoring systems aimed at supporting inclusive and effective STEAM education in higher education contexts.