To read this content please select one of the options below:

Understanding and detecting data fabrication in large-scale assessments

Kentaro Yamamoto (Educational Testing Service, Princeton, New Jersey, USA)
Mary Louise Lennon (Educational Testing Service, Princeton, New Jersey, USA)

Quality Assurance in Education

ISSN: 0968-4883

Article publication date: 3 April 2018

467

Abstract

Purpose

Fabricated data jeopardize the reliability of large-scale population surveys and reduce the comparability of such efforts by destroying the linkage between data and measurement constructs. Such data result in the loss of comparability across participating countries and, in the case of cyclical surveys, between past and present surveys. This paper aims to describe how data fabrication can be understood in the context of the complex processes involved in the collection, handling, submission and analysis of large-scale assessment data. The actors involved in those processes, and their possible motivations for data fabrication, are also elaborated.

Design/methodology/approach

Computer-based assessments produce new types of information that enable us to detect the possibility of data fabrication, and therefore the need for further investigation and analysis. The paper presents three examples that illustrate how data fabrication was identified and documented in the Programme for the International Assessment of Adult Competencies (PIAAC) and the Programme for International Student Assessment (PISA) and discusses the resulting remediation efforts.

Findings

For two countries that participated in the first round of PIAAC, the data showed a subset of interviewers who handled many more cases than others. In Case 1, the average proficiency for respondents in those interviewers’ caseloads was much higher than expected and included many duplicate response patterns. In Case 2, anomalous response patterns were identified. Case 3 presents findings based on data analyses for one PISA country, where results for human-coded responses were shown to be highly inflated compared to past results.

Originality/value

This paper shows how new sources of data, such as timing information collected in computer-based assessments, can be combined with other traditional sources to detect fabrication.

Keywords

Citation

Yamamoto, K. and Lennon, M.L. (2018), "Understanding and detecting data fabrication in large-scale assessments", Quality Assurance in Education, Vol. 26 No. 2, pp. 196-212. https://doi.org/10.1108/QAE-07-2017-0038

Publisher

:

Emerald Publishing Limited

Copyright © 2018, Emerald Publishing Limited

Related articles