Search results

This paper aims to provides a brief review of the dashboard literature, an account of the development of performance dashboards for field data collection at Westat, and more specifically for the first cycle of the Programme for the International Assessment of Adult Competencies (PIAAC). A brief concluding section offers suggestions for improvements in survey dashboards for PIAAC’s next cycle.

To manage field work at Westat, the authors create views into various paradata databases and present them in a dashboard, showing key performance indicators at a glance. Users can drill down from the dashboard into underlying databases to investigate potential problems. The US PIAAC dashboard is a monitoring system that supports daily review of many activities. For example, it provides overnight alerts to the field supervisor when global positioning system (GPS) data from an interviewer’s smartphone shows the interview occurred far from the respondent’s home.

Performance dashboards may represent best practice for monitoring field activities. Paradata sources and systems vary greatly across the PIAAC countries, but a multitude of process data exists in every country and can be used to create quality indicators and a monitoring system. PIAAC can establish standards/guidelines to improve visualization of quality metrics and management data, regardless of the local survey infrastructure.

The core of the paper is a case study of the experiences on the US PIAAC implementation of dashboards to monitor survey quality, production and costs, with special attention to the issue of fabrication.

Fabricated data jeopardize the reliability of large-scale population surveys and reduce the comparability of such efforts by destroying the linkage between data and measurement constructs. Such data result in the loss of comparability across participating countries and, in the case of cyclical surveys, between past and present surveys. This paper aims to describe how data fabrication can be understood in the context of the complex processes involved in the collection, handling, submission and analysis of large-scale assessment data. The actors involved in those processes, and their possible motivations for data fabrication, are also elaborated.

Computer-based assessments produce new types of information that enable us to detect the possibility of data fabrication, and therefore the need for further investigation and analysis. The paper presents three examples that illustrate how data fabrication was identified and documented in the Programme for the International Assessment of Adult Competencies (PIAAC) and the Programme for International Student Assessment (PISA) and discusses the resulting remediation efforts.

For two countries that participated in the first round of PIAAC, the data showed a subset of interviewers who handled many more cases than others. In Case 1, the average proficiency for respondents in those interviewers’ caseloads was much higher than expected and included many duplicate response patterns. In Case 2, anomalous response patterns were identified. Case 3 presents findings based on data analyses for one PISA country, where results for human-coded responses were shown to be highly inflated compared to past results.

This paper shows how new sources of data, such as timing information collected in computer-based assessments, can be combined with other traditional sources to detect fabrication.

Research has shown that the problem-solving skills of adults with a vocational education and training (VET) background in technology-rich environments (TREs) are often inadequate. However, some adults with a VET background do have sound problem-solving skills. The present study aims to provide insight into the socio-demographic, work-related and everyday life factors that are associated with a strong problem-solving performance.

The study builds on large-scale data of the Programme for the International Assessment of Adult Competencies (PIAAC) and gives insight into VET adults (N = 12,929) with strong problem-solving skills in 11 European countries.

This study introduces new knowledge with respect to the socio-demographic, work-related and everyday life background factors that contribute to successful VET adults’ problem-solving skills. The findings of the authors illustrate that a continuous process of development including non-formal and informal activity, as well as learning taking place at work, is associated with strong performance in problem-solving skills in TRE.

An important implication of this study is that this paper introduces novel knowledge for VET adults’ competences and can be used to support the development of VET adults’ problem-solving skills in TREs.

The study was conducted to explore new understanding about good problem-solvers in TREs with a VET background. The originality of the study derives from its focus on good problem-solvers in TREs related to a VET background. The findings can be used to create novel ways to enhance the development of VET adults’ problem-solving skills in TREs.

This paper aims to investigate small-scale, qualitative observations of interviewer–respondent interaction in the Organisation for Economic Co-operation and Development’s Programme for the International Assessment of Adult Competencies (PIAAC).

The paper uses video-ethnographic methods to document talk and gesture in assessment in Slovenian household settings. It presents an in-depth case study of interaction in a single testing situation.

Observing interaction in assessment captures data on assessment performance that is not available in quantitative analysis of assessment response processes. The character of interviewer–respondent interaction and rapport is shaped by the cognitive demands of assessment and the distinctive ecological setting of the household.

Observational data on assessment response processes and interaction in real-life assessments can be integrated into and synthesized with other sources of “process data”.

Assessment programs such as PIAAC should consider the significance of the household setting on assessment quality and observations of interaction in assessment as a valid source of paradata.

There is a place for small-scale observational studies of assessment to inform public understanding of assessment quality and validity.

The paper provides qualitative insights into the significance of interaction and “interviewer effects” in household assessment settings.

The purpose of this paper is to examine the relationships between education and competencies from a comparative view, while controlling for background characteristics (e.g. parental education), home features (e.g. income) and country fixed effect.

This paper utilises an international data sets of 12 Belt and Road (B&R) countries participating in the PIAAC survey. Data are examined using regression models with “REPSET” Stata code. The focus on B&R countries is interesting as this recent economic and strategic development alliance might transform the contemporary global balance of power. Therefore, examining its educational development is important.

Findings reveal the higher the level of education, the higher the competency level in numeracy and literacy. Age was found to be negatively related to competencies, and gender was found to be dually significant. Among the examined B&R countries, Slovenia, Russia and Israel contribute the most to the numeracy, literacy and problem solving competencies (respectively) compared with other countries. The findings might contribute to the design of education for development policies aimed at increasing the level of state competitiveness while accounting for social cohesiveness.

The paper provides a comparative study into educational development of the B&R alliance, and therefore offers an insight of the developmental cooperation as it emerges. The focus on the B&R initiative is important as it is a strategic development alliance which might transform the contemporary global balance of power. Therefore, the examination of the above-mentioned relationships with the focus on B&R countries might advance the strategic design of this cooperation.

Surveys in multinational, multiregional and multicultural contexts (or “3MC” surveys) are becoming increasingly important to global and regional decision-making and theory building. To serve this purpose, the surveys need to be well managed, with an awareness of key sources of survey error and how to minimize them, mechanisms in place to control the implementation process and an ability to intervene in that process when necessary in a spirit of continuous improvement (Pennell et al., 2017). One key approach for managing and assessing the quality of 3MC surveys is the total survey error (TSE) framework and associated survey process quality. This paper aims to examine the application of the TSE framework and survey process quality to the Programme for the International Assessment of Adult Competencies (PIAAC).

The authors begin with a background on TSE and discuss recent adaptations of TSE and survey process quality for 3MC surveys. They then presents a TSE framework tailored with examples of potential contributions to error for PIAAC and ways to address those through effective quality assurance (QA) and quality control (QC) approaches.

Overall, the authors find that the design and implementation of the first cycle of PIAAC largely reflect the current best practice for 3MC surveys. However, the authors identify several potential contributions to error that may threaten comparability in PIAAC and ways these could be addressed in the upcoming cycle.

With a view toward continuous improvement, the final section draws on the survey process quality approach adapted from Hansen et al.’s study (2016) to summarize the recommendations in terms of additional QA elements (inputs and activities) and associated QC elements (measures and reports) for PIAAC’s consideration in the next cycle.

This research explores how skill proficiencies are distributed between low-income and not-in low-income groups using the results of a highly complex survey of the information-processing skills of Canadians between the ages of 16 and 65. We find that having measures of skills enhances our understanding of the correlates of low income. Skills have an independent effect, even when controlling for other known correlates of low income, and their inclusion reduces the independent effect of education and immigrant status. This result is relevant for public policy development as the knowledge of the skills profile of the low-income population can inform the design of efficient and effective programmes.

Utilizing datasets of Ecuador, Hungary, Kazakhstan, Mexico and Peru from the Programme for the International Assessment of Adult Competencies survey from 2017 to 2018, this study aimed to develop and validate a profile indicating core workplace skills in developing countries.

DeVellis' guide of scale development navigated the development of the profile. Multiple techniques including item analysis, exploratory factor analysis, confirmatory factor analysis and multigroup confirmatory factor analysis were used on a sample of 7,166 participants to validate the profile of core workplace skills in developing countries.

A resultant five-dimensional profile with 18 items was developed: oral communication skills, reading skills, math skills, information and communication technology skills and learning skills. The estimates of composite reliability showed the profile was reliable. The validity estimates of the profile were obtained from several sources including content, convergent, discriminative and construct validity. The measurement invariance was also held for the profile.

Based on the researcher's knowledge, the study is the first attempt to develop a profile to indicate core workplace skills in developing countries. The profile theoretically framed the core workplace skills in developing countries and provides a new measure for identifying, evaluating and thus improving core workplace skills in developing countries for different stakeholders in the era of Education 4.0.

Surveys that include skill measures may suffer from additional sources of error compared to those containing questionnaires alone. Examples are distractions such as noise or interruptions of testing sessions, as well as fatigue or lack of motivation to succeed. This paper aims to provide a review of statistical tools based on latent variable modeling approaches extended by explanatory variables that allow detection of survey errors in skill surveys.

This paper reviews psychometric methods for detecting sources of error in cognitive assessments and questionnaires. Aside from traditional item responses, new sources of data in computer-based assessment are available – timing data from the Programme for the International Assessment of Adult Competencies (PIAAC) and data from questionnaires – to help detect survey errors.

Some unexpected results are reported. Respondents who tend to use response sets have lower expected values on PIAAC literacy scales, even after controlling for scores on the skill-use scale that was used to derive the response tendency.

The use of new sources of data, such as timing and log-file or process data information, provides new avenues to detect response errors. It demonstrates that large data collections need to better utilize available information and that integration of assessment, modeling and substantive theory needs to be taken more seriously.

In a world where the continual combining of computer applications and the expansion of artificial intelligence is already necessarily changing the world of work for people, an education system that does not adequately respond to these trends and changes will render itself irrelevant. Education policy and regulation may suffer at the hand of such accelerations due to unexpected consequences and developments. However, the rapid, exponential improvements in computer hardware and software that have enhanced the rate and our ability to gather, transform, manipulate, and interpret these data in an ongoing fashion also present myriad educational opportunities. The so-called Fourth Industrial Revolution offers societies data and information capabilities previously unimagined, making it possible to learn how to combine, innovate, and imagine entirely new avenues for building responsive and intelligent education policies and systems that promote the education and wellbeing of citizens as well as improving their economic participation. These advances necessitate a growing number of educators and education systems who can intelligently respond to Industry 4.0 trends. In this chapter, some considerations regarding the use of large-scale, international datasets and emerging data analytics for analyzing policy for the governance of education are offered, and a discussion of the need for the more systematic use of data analytics as a mechanism for developing socially responsive adult learning and workforce education policy and programing.