Search results

This AERI-NEPC eBrief, the fourth in a series titled “Understanding validity issues around the world”, looks closely at issues surrounding the validity of test-based actions in educational accountability and school improvement contexts. The specific discussions here focus testing issues in the US. However, the general principles underlying appropriate and inappropriate test use in school reform and high stakes public accountability settings are applicable in both domestic and international settings. This paper aims to present the issues.

This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. It begins by summarizing perspectives of an invited expert panel on the topic. To that synthesis, the authors add their own analysis of key issues. They conclude by offering recommendations for test developers and test users.

The authors conclude that recurring validity issues arise with tests used in school reform and public accountability contexts, because the external tests tend to be employed as policy instruments to drive reforms in schools, with unrealistic timelines and inadequate resources. To reconcile the validity issues with respect to educational assessment and forge a coherent understanding of validity among multiple public users with different agendas, the authors offer several recommendations, such as: adopt an integrated approach to develop content and standards of proficiency that represent a range of cognitive processes; support studies to examine validity of assessments and the effects of decisions taken with assessment data before results are fed into high stakes accountability-related actions that affect teachers, leaders or schools; align standards, curricula, instruction, assessment, and professional development efforts in schools to maximize success; increase capacity-building efforts to help teachers, administrators, policy makers, and other groups of test users learn more about assessments, particularly, about appropriate interpretation and use of assessment data and reports.

Baker points out that in response to growing demands of reformers and policy-makers for more frequent and rigorous testing programs in US public education, results from a single test tend to get used to meet a variety of public education needs today (e.g. school accountability, school improvement, teacher evaluation, and measurement of student performance). While this may simply be a way to make things more cost-efficient and reduce the extent of student testing in schools, a consequence is inappropriate test use that threatens validity in practice settings. This policy brief confronts this recurring validity challenge and offers recommendations to address the issues.

This policy brief discusses validity and fairness issues that could arise when test-based information is used for making “high stakes” decisions at an individual level, such as, for the certification of teachers or other professionals, or when admitting students into higher education programs and colleges, or for making immigration-related decisions for prospective immigrants. To assist test developers, affiliated researchers and test users enhance levels of validity and fairness with these particular types of test score interpretations and uses, this policy brief summarizes an “argument-based approach” to validation given by Kane.

This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. To that synthesis, the authors add practitioner-friendly examples with their own analysis of key issues. They conclude by offering recommendations for test developers and test users.

The authors conclude that validity is a complex and evolving construct, especially when considering issues of fairness in individual testing contexts. Kane's argument-based approach offers an accessible framework through which test makers can accumulate evidence to evaluate inferences and arguments related to decisions to be made with test scores. Perspectives of test makers, researchers, test takers and decision-makers must all be incorporated into constructing coherent “validity arguments” to guide the test development and validation processes.

Standardized test use for individual-level decisions is gradually spreading to various regions of the world, but understandings of validity are still uneven among key stakeholders of such testing programs. By translating complex information on test validation, validity and fairness issues with all concerned stakeholders in mind, this policy brief attempts to address the communication gaps noted to exist among these groups by Kane.

The aim of this research is to investigate the consistency between the criterion validity and the unidimensional validity – case study in management research.

The investigation of construct validity and criterion validity by using confirmatory factor analysis is presented in this paper. Data were found from three dissertations of Doctoral Program of Management on Brawijaya University and Airlangga University. It consists of seven variables. The first data consist of three variables, i.e. X1, X2 and Y1; the second data consist of two variables, i.e. X2 and Y1; and the third data consist of two variables, i.e. X3 and Y1.

The measurement instrument of latent variables must be valid. There are many methods of construct validity, i.e. unidimensional validity and criterion validity. The criterion validity is less sensitive than unidimensional validity, that is, it is more simple and easy for application.

The research is limited to only analyzing criterion validity and unidimensional validity.

The unidimensional validity is more sensitive than criterion validity, that is, it is more complex and difficult for application. But if structural equation modeling (SEM) is used to analyze data, the unidimensional validity test must be used; this causes SEM to have constraint, i.e. factor indeterminacy.

Majority in Dissertation of Doctoral Program (Brawijaya University and Airlangga University) using criterion validity to investigate the instrument validity, without investigating the unidimensional validity, is included in this paper.

To investigate the instrument validity, no studies have focused on the selection of instrument validity, particularly in the field of management.

The purpose of this paper is to review and extend recent simulation studies on discriminant validity measures, contrasting the use of cutoff values (i.e. heuristics) with inferential tests.

Based on a simulation study, which considers different construct correlations, sample sizes, numbers of indicators and loading patterns, the authors assess each criterion’s sensitivity to type I and type II errors.

The findings of the simulation study provide further evidence for the robustness of the heterotrait–monotrait (HTMT) ratio of correlations criterion as an estimator of disattenuated (perfectly reliable) correlations between constructs, whose performance parallels that of the standard constrained PHI approach. Furthermore, the authors identify situations in which both methods fail and suggest an alternative criterion.

Addressing the limitations of prior simulation studies, the authors use both directional comparisons (i.e. heuristics) and inferential tests to facilitate the comparison of the HTMT and PHI methods. Furthermore, the simulation considers criteria that have not been assessed in prior research.

The purpose of this paper, a critical reflection, is twofold. First, by comprehensively reviewing scale development procedures in hospitality research, a concerning lack of nomological validity testing is demonstrated. Second, the need for nomological validity testing is discussed and both conceptually and empirically reasoned.

This research systematically reviews scale development studies in three leading hospitality journals, including Cornell Hospitality Quarterly, International Journal of Contemporary Hospitality Management and International Journal of Hospitality Management over ten years (2012–2021) to analyze the completeness of scale development procedures. Specifically, the authors evaluate whether the reviewed studies engage in testing the nomological and predictive validity of the newly developed measures.

The results indicate a concerning gap in the current practices in hospitality research. Specifically, only 33.3% of the examined studies assess nomological validity. These findings collectively underscore the need for improving the comprehensiveness of scale development processes in hospitality research.

The study offers important implications for hospitality researchers. The paper provides an extensive discussion on the importance and benefits of testing for nomological validity in scale development studies, contributing to the completeness and consistency of scale development procedures in the hospitality discipline.

This research critically assesses prevalent, and widely accepted, scale development procedures in hospitality research. This research empirically demonstrates the neglect of nomological validity issues in scale development practices in hospitality research. Scale development is an essential scientific practice used to create a research instrument in a field of study, improving our understanding of a specific phenomenon and contributing to knowledge creation. Considering the significance of scale development in advancing the field of hospitality research, the validation procedures involved in the scale development processes are of utmost importance and should be thoroughly applied.

Auditing researchers have recommended that the use of audit decision models should be subject to cost‐benefit analysis. This paper provides insight into cost‐benefit analysis and its shortcomings as a tool for evaluating audit decision models. The paper also identifies and discusses the limitations of other evaluation methods. Finally, the paper suggests the use of model confidence as an alternative to model value and model validity.

Addresses the standardization of the measurements and the labels for concepts commonly used in the study of work organizations. As a reference handbook and research tool, seeks to improve measurement in the study of work organizations and to facilitate the teaching of introductory courses in this subject. Focuses solely on work organizations, that is, social systems in which members work for money. Defines measurement and distinguishes four levels: nominal, ordinal, interval and ratio. Selects specific measures on the basis of quality, diversity, simplicity and availability and evaluates each measure for its validity and reliability. Employs a set of 38 concepts ‐ ranging from “absenteeism” to “turnover” as the handbook’s frame of reference. Concludes by reviewing organizational measurement over the past 30 years and recommending future measurement reseach.