Search results

Surveys in multinational, multiregional and multicultural contexts (or “3MC” surveys) are becoming increasingly important to global and regional decision-making and theory building. To serve this purpose, the surveys need to be well managed, with an awareness of key sources of survey error and how to minimize them, mechanisms in place to control the implementation process and an ability to intervene in that process when necessary in a spirit of continuous improvement (Pennell et al., 2017). One key approach for managing and assessing the quality of 3MC surveys is the total survey error (TSE) framework and associated survey process quality. This paper aims to examine the application of the TSE framework and survey process quality to the Programme for the International Assessment of Adult Competencies (PIAAC).

The authors begin with a background on TSE and discuss recent adaptations of TSE and survey process quality for 3MC surveys. They then presents a TSE framework tailored with examples of potential contributions to error for PIAAC and ways to address those through effective quality assurance (QA) and quality control (QC) approaches.

Overall, the authors find that the design and implementation of the first cycle of PIAAC largely reflect the current best practice for 3MC surveys. However, the authors identify several potential contributions to error that may threaten comparability in PIAAC and ways these could be addressed in the upcoming cycle.

With a view toward continuous improvement, the final section draws on the survey process quality approach adapted from Hansen et al.’s study (2016) to summarize the recommendations in terms of additional QA elements (inputs and activities) and associated QC elements (measures and reports) for PIAAC’s consideration in the next cycle.

This paper aims to provides a brief review of the dashboard literature, an account of the development of performance dashboards for field data collection at Westat, and more specifically for the first cycle of the Programme for the International Assessment of Adult Competencies (PIAAC). A brief concluding section offers suggestions for improvements in survey dashboards for PIAAC’s next cycle.

To manage field work at Westat, the authors create views into various paradata databases and present them in a dashboard, showing key performance indicators at a glance. Users can drill down from the dashboard into underlying databases to investigate potential problems. The US PIAAC dashboard is a monitoring system that supports daily review of many activities. For example, it provides overnight alerts to the field supervisor when global positioning system (GPS) data from an interviewer’s smartphone shows the interview occurred far from the respondent’s home.

Performance dashboards may represent best practice for monitoring field activities. Paradata sources and systems vary greatly across the PIAAC countries, but a multitude of process data exists in every country and can be used to create quality indicators and a monitoring system. PIAAC can establish standards/guidelines to improve visualization of quality metrics and management data, regardless of the local survey infrastructure.

The core of the paper is a case study of the experiences on the US PIAAC implementation of dashboards to monitor survey quality, production and costs, with special attention to the issue of fabrication.

This paper aims to investigate small-scale, qualitative observations of interviewer–respondent interaction in the Organisation for Economic Co-operation and Development’s Programme for the International Assessment of Adult Competencies (PIAAC).

The paper uses video-ethnographic methods to document talk and gesture in assessment in Slovenian household settings. It presents an in-depth case study of interaction in a single testing situation.

Observing interaction in assessment captures data on assessment performance that is not available in quantitative analysis of assessment response processes. The character of interviewer–respondent interaction and rapport is shaped by the cognitive demands of assessment and the distinctive ecological setting of the household.

Observational data on assessment response processes and interaction in real-life assessments can be integrated into and synthesized with other sources of “process data”.

Assessment programs such as PIAAC should consider the significance of the household setting on assessment quality and observations of interaction in assessment as a valid source of paradata.

There is a place for small-scale observational studies of assessment to inform public understanding of assessment quality and validity.

The paper provides qualitative insights into the significance of interaction and “interviewer effects” in household assessment settings.

The purpose of this paper is to examine the relationships between education and competencies from a comparative view, while controlling for background characteristics (e.g. parental education), home features (e.g. income) and country fixed effect.

This paper utilises an international data sets of 12 Belt and Road (B&R) countries participating in the PIAAC survey. Data are examined using regression models with “REPSET” Stata code. The focus on B&R countries is interesting as this recent economic and strategic development alliance might transform the contemporary global balance of power. Therefore, examining its educational development is important.

Findings reveal the higher the level of education, the higher the competency level in numeracy and literacy. Age was found to be negatively related to competencies, and gender was found to be dually significant. Among the examined B&R countries, Slovenia, Russia and Israel contribute the most to the numeracy, literacy and problem solving competencies (respectively) compared with other countries. The findings might contribute to the design of education for development policies aimed at increasing the level of state competitiveness while accounting for social cohesiveness.

The paper provides a comparative study into educational development of the B&R alliance, and therefore offers an insight of the developmental cooperation as it emerges. The focus on the B&R initiative is important as it is a strategic development alliance which might transform the contemporary global balance of power. Therefore, the examination of the above-mentioned relationships with the focus on B&R countries might advance the strategic design of this cooperation.

This research explores how skill proficiencies are distributed between low-income and not-in low-income groups using the results of a highly complex survey of the information-processing skills of Canadians between the ages of 16 and 65. We find that having measures of skills enhances our understanding of the correlates of low income. Skills have an independent effect, even when controlling for other known correlates of low income, and their inclusion reduces the independent effect of education and immigrant status. This result is relevant for public policy development as the knowledge of the skills profile of the low-income population can inform the design of efficient and effective programmes.

Utilizing datasets of Ecuador, Hungary, Kazakhstan, Mexico and Peru from the Programme for the International Assessment of Adult Competencies survey from 2017 to 2018, this study aimed to develop and validate a profile indicating core workplace skills in developing countries.

DeVellis' guide of scale development navigated the development of the profile. Multiple techniques including item analysis, exploratory factor analysis, confirmatory factor analysis and multigroup confirmatory factor analysis were used on a sample of 7,166 participants to validate the profile of core workplace skills in developing countries.

A resultant five-dimensional profile with 18 items was developed: oral communication skills, reading skills, math skills, information and communication technology skills and learning skills. The estimates of composite reliability showed the profile was reliable. The validity estimates of the profile were obtained from several sources including content, convergent, discriminative and construct validity. The measurement invariance was also held for the profile.

Based on the researcher's knowledge, the study is the first attempt to develop a profile to indicate core workplace skills in developing countries. The profile theoretically framed the core workplace skills in developing countries and provides a new measure for identifying, evaluating and thus improving core workplace skills in developing countries for different stakeholders in the era of Education 4.0.

Fabricated data jeopardize the reliability of large-scale population surveys and reduce the comparability of such efforts by destroying the linkage between data and measurement constructs. Such data result in the loss of comparability across participating countries and, in the case of cyclical surveys, between past and present surveys. This paper aims to describe how data fabrication can be understood in the context of the complex processes involved in the collection, handling, submission and analysis of large-scale assessment data. The actors involved in those processes, and their possible motivations for data fabrication, are also elaborated.

Computer-based assessments produce new types of information that enable us to detect the possibility of data fabrication, and therefore the need for further investigation and analysis. The paper presents three examples that illustrate how data fabrication was identified and documented in the Programme for the International Assessment of Adult Competencies (PIAAC) and the Programme for International Student Assessment (PISA) and discusses the resulting remediation efforts.

For two countries that participated in the first round of PIAAC, the data showed a subset of interviewers who handled many more cases than others. In Case 1, the average proficiency for respondents in those interviewers’ caseloads was much higher than expected and included many duplicate response patterns. In Case 2, anomalous response patterns were identified. Case 3 presents findings based on data analyses for one PISA country, where results for human-coded responses were shown to be highly inflated compared to past results.

This paper shows how new sources of data, such as timing information collected in computer-based assessments, can be combined with other traditional sources to detect fabrication.

Research has shown that the problem-solving skills of adults with a vocational education and training (VET) background in technology-rich environments (TREs) are often inadequate. However, some adults with a VET background do have sound problem-solving skills. The present study aims to provide insight into the socio-demographic, work-related and everyday life factors that are associated with a strong problem-solving performance.

The study builds on large-scale data of the Programme for the International Assessment of Adult Competencies (PIAAC) and gives insight into VET adults (N = 12,929) with strong problem-solving skills in 11 European countries.

This study introduces new knowledge with respect to the socio-demographic, work-related and everyday life background factors that contribute to successful VET adults’ problem-solving skills. The findings of the authors illustrate that a continuous process of development including non-formal and informal activity, as well as learning taking place at work, is associated with strong performance in problem-solving skills in TRE.

An important implication of this study is that this paper introduces novel knowledge for VET adults’ competences and can be used to support the development of VET adults’ problem-solving skills in TREs.

The study was conducted to explore new understanding about good problem-solvers in TREs with a VET background. The originality of the study derives from its focus on good problem-solvers in TREs related to a VET background. The findings can be used to create novel ways to enhance the development of VET adults’ problem-solving skills in TREs.

At the invitation of the Programme for the International Assessment of Adult Competencies (PIAAC), this paper aims to examine advances in survey interviewer monitoring and make recommendations on minimizing the effect of interviewers on survey results.

The authors first provide an overview of the most recent literature on interviewer effects, quality assurance and quality control. Here, they draw upon recent publications such as the cross-cultural survey guidelines (www.ccsg.isr.umich.edu) and newly published or in-press material specifically addressing these issues in multicultural, multinational and multiregional (3MC) contexts.

The authors discuss trends and innovations in quality assurance and quality control in 3MC studies and draw upon examples from international surveys that are using cutting-edge and innovative approaches to monitor interviewer behavior and minimize interviewer effects.

With a view to continuous quality improvement, the authors conclude with concrete recommendations for PIAAC to consider for the next cycle. Many of the recommendations have general relevance for other large-scale cross-national surveys.

This paper aims to find a measurement model with better evidence of validity, with data extracted from the Program for the International Assessment of Adult Competencies (PIAAC). To test a parsimonious model in which dispositional and workplace context characteristics are predictors of informal learning behaviors (ILBs).

The authors performed exploratory and confirmatory factor analyses to improve the fit of the PIAAC data measurement model. Multiple linear regression was used to examine the prediction of ILBs by one dispositional variable (Readiness to Learn) and two workplace context variables (Autonomy and Interaction in the Workplace).

A measurement model emerged with 18 items divided into four factors. The three antecedent variables predicted ILBs. Interaction in the workplace resulted in higher scores, and workplace autonomy resulted in lower scores.

The small number of items for ILBs prevented a more detailed exploration of predictors of different types of these behaviors. ILBs can be stimulated by policies that promote readiness to learn and that encourage the design of environments that require worker interactions and autonomy.

Few studies on ILBs in the workplace have investigated the prediction of dispositional and contextual antecedents based on a theoretical model. The findings herein were obtained using a diverse sample of countries, occupations and generations, allowing better generalization. The importance of interpersonal relationships in the workplace for predicting ILBs was emphasized.