Moderating Possibly Irrelevant Multiple Mean Score Differences on a Test of Mathematical Reasoning

A pool of items from operational tests of mathematical reasoning was constructed to investigate the feasibility of using automated test assembly (ATA) methods to simultaneously moderate possibly irrelevant differences between the performance of women and men, and African American and White test takers. None of the artificial tests exhibited substantial impact moderation, although the estimated mean scaled score differences for the relevant population indicated a modest move in the intended direction: the difference between scaled score means was reduced by about 20% for women and men and about 9% for African American and White test takers. Although many issues in the implementation of this methodology remain to be solved, the consideration of impact in ATA, along with the maintenance of the detailed test plan, appears to be a potential method of moderating possibly irrelevant mean test score differences.