Evaluating the validity and comparability of PIRLS 2016 in South Africa

Large-scale assessments of reading, such as the Progress in International Reading Literacy Study (PIRLS), are useful for understanding reading development. PIRLS aims to provide valid and reliable assessments of reading across multiple countries. However, the extreme range in reading achievement outcomes, from the very high performance of some countries to the very low performance of others, creates substantial disparities across countries that threaten the validity, reliability, and fairness of PIRLS results. South Africa participates in PIRLS to better understand and inform interventions that can help improve reading in the country. But consistently low performance in PIRLS, coupled with intricate multilingualism, historical racial discrimination, and extreme inequality present further challenges to the already complicated task of assessing reading. This thesis evaluates the validity and comparability of PIRLS 2016 in the unique local context of South Africa. It considers the implications of factors such as low overall performance, performance differences across groups, fit of the analytical models used, and comparability of item difficulty across language versions for the validity of the test and items. Because of the complexities associated with large-scale assessment in diverse contexts, the thesis brings together three papers, each focused on a specific area of concern regarding PIRLS 2016 in South Africa. While highlighting some concerns, the papers also discuss the implications that factors such as low performance and language diversity have for the valid interpretation of PIRLS scores, as well as the implications for the use of PIRLS to inform education policy and practice in South Africa. The first paper examines the relationship between test language and reading achievement, revealing that test language significantly influences overall reading achievement in PIRLS 2016 in South Africa, even when controlling for other influential contextual factors. The second paper evaluates the validity of the PIRLS 2016 test instrument for South African students and highlights substantial issues with the alignment between item difficulty and student ability. The third paper investigates comparability across language versions and uncovers substantial evidence of comparability concerns at the item and passage level. Overall, the findings from this thesis raise concerns about the use of PIRLS data to inform educational policy in South Africa. By highlighting key challenges in the South African context, this thesis contributes to a deeper understanding of the importance of addressing the needs of unique local contexts to ensure large-scale assessments are able to provide valid, reliable, and fair information for all participating countries.