Understanding validity issues in international large scale assessments

Meiko Lin (Interdisciplinary Studies, Teachers College, Columbia University, New York, New York, USA)
Erin Bumgarner (Tufts Interdisciplinary Evaluation Research, Tufts University, Medford, Massachusetts, USA)
Madhabi Chatterji (Organization and Leadership, Teachers College, Columbia University, New York, New York, USA)

Quality Assurance in Education

ISSN: 0968-4883

Publication date: 28 January 2014

Abstract

Purpose

This policy brief, the third in the AERI-NEPC eBrief series “Understanding validity issues around the world”, discusses validity issues surrounding International Large Scale Assessment (ILSA) programs. ILSA programs, such as the well-known Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), are rapidly expanding around the world today. In this eBrief, the authors examine what “validity” means when applied to published results and reports of programs like the PISA.

Design/methodology/approach

This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. It begins by summarizing perspectives of an invited expert panel on the topic. To that synthesis, the authors add their own analysis of key issues. They conclude by offering recommendations for test developers and test users.

Findings

ILSA programs and tests, while offering valuable information, should be read and used cautiously and in context. All parties need to be on the same page to maximize valid use of ILSA results, to obtain the greatest educational and social benefits, and to minimize negative consequences. The authors propose several recommendations for test makers and ILSA program leaders, and ILSA users. To ILSA leaders and researchers: provide more cautionary information about how to correctly interpret the ILSA results, particularly country rankings, given contextual differences among nations. Provide continuing psychometric or research resources so as to address or reduce various sources of error in reports. Encourage policy makers in different nations to share the responsibility for ensuring more contextualized (and valid) interpretations of ILSA reports and subsequent policy development. Raise awareness among policy makers to look beyond simple rankings and pay more attention to inter-country differences. For consumers of ILSA results and reports: read the fine print, not just the country rankings, to interpret ILSA results correctly in particular regions/nations. When looking to high-ranking countries as role models, be sure to consider the “whole picture”. Use ILSA data as complements to other national- and state-level educational assessments to better gauge the status of the country's education system and subsequent policy directions.

Originality/value

By translating complex information on validity issues with all concerned ILSA stakeholders in mind, this policy brief will improve uses and applications of ILSA information in national and regional policy contexts.

Keywords

Citation

Lin, M., Bumgarner, E. and Chatterji, M. (2014), "Understanding validity issues in international large scale assessments", Quality Assurance in Education, Vol. 22 No. 1, pp. 31-41. https://doi.org/10.1108/QAE-12-2013-0050

Download as .RIS

Publisher

:

Emerald Group Publishing Limited

Copyright © 2014, Authors

Please note you might not have access to this content

You may be able to access this content by login via Shibboleth, Open Athens or with your Emerald account.
If you would like to contact us about accessing this content, click the button and fill out the form.
To rent this content from Deepdyve, please click the button.