Assessing Early Literacy at Scale: Format Effects in Unsupervised Digital Assessment

Assessing foundational literacy at scale in low-resource settings is challenging due to the cost of assessor-led testing. Digital platforms offer a scalable alternative, but the performance of different item formats under unsupervised conditions, particularly for young learners, remains under-evidenced. This paper analyses a large-scale deployment of digital assessments delivered through EIDU's platform-based education programme in pre-primary classrooms in Kenya, comprising multiple-choice questions (MCQs) and very short answer questions (VSAQs). Across 230,000 learners and 1.8 million item responses, MCQs achieved significantly higher completion and scores than VSAQs, with item format explaining 31\% of variance in completed-assessment scores. However, clustering incorrect VSAQ responses revealed highly structured patterns of partial knowledge, response-format confusion, and emerging phonics understanding that binary scoring entirely obscures. Binary correctness scoring of both MCQs and VSAQs showed no significant association with an independent literacy assessment (IDELA; linked sample n = 384), but classifying incorrect VSAQ responses by degree of understanding uncovered a significant directional signal, though whether this generalises beyond the small validation sample remains an open question. The results highlight a trade-off between MCQ engagement and the richer learner signal in VSAQ errors, motivating further validation of their diagnostic value.