Paradata and dashboards in PIAAC

Abstract

Purpose

This paper aims to provides a brief review of the dashboard literature, an account of the development of performance dashboards for field data collection at Westat, and more specifically for the first cycle of the Programme for the International Assessment of Adult Competencies (PIAAC). A brief concluding section offers suggestions for improvements in survey dashboards for PIAAC’s next cycle.

Design/methodology/approach

To manage field work at Westat, the authors create views into various paradata databases and present them in a dashboard, showing key performance indicators at a glance. Users can drill down from the dashboard into underlying databases to investigate potential problems. The US PIAAC dashboard is a monitoring system that supports daily review of many activities. For example, it provides overnight alerts to the field supervisor when global positioning system (GPS) data from an interviewer’s smartphone shows the interview occurred far from the respondent’s home.

Findings

Performance dashboards may represent best practice for monitoring field activities. Paradata sources and systems vary greatly across the PIAAC countries, but a multitude of process data exists in every country and can be used to create quality indicators and a monitoring system. PIAAC can establish standards/guidelines to improve visualization of quality metrics and management data, regardless of the local survey infrastructure.

Originality/value

The core of the paper is a case study of the experiences on the US PIAAC implementation of dashboards to monitor survey quality, production and costs, with special attention to the issue of fabrication.

Citation

Mohadjer, L. and Edwards, B. (2018), "Paradata and dashboards in PIAAC", Quality Assurance in Education, Vol. 26 No. 2, pp. 263-277. https://doi.org/10.1108/QAE-06-2017-0031