Search results

This chapter, completed in 1999, provides an overview and critical analysis of the validity research agenda undertaken by the National Board for Professional Teaching Standards (NBPTS) for its assessment to certify accomplished teachers at the end of its first decade of assessment development and implementation. The review is presented in three major sections: (a) an overview of the validity criteria underlying the review; (b) a description of the National Board's research agenda presented in its own terms, focusing first on the studies that were routinely carried out for each certificate and second on the “special studies” that were not part of the routine agenda; and (c) a series of six critical observations and explanations based on the validity issues described in the first section.

The purpose of this paper is to validate empirically a measurement scale of the effectiveness of R & D activities, starting from previous work in which the content was validated.

Following psychometric standards the authors have addressed the analysis phases of construct dimensionality, reliability and validity (convergent, discriminant and nomologic), and the scale criteria are shown to be valid in their three temporal manifestations (retrospective, concurrent and predictive). The empirical evidence was drawn from a sample of 85 companies belonging to the Spanish pharmaceutical sector.

Globally the authors provide evidence of reliability, validity of construct and validity of criterion in their diverse manifestations, for the scale designed and validated, on effectiveness in R & D. The authors divide the results into two groups: one for content of the scale and the other for relationships of the scale with other variables. With respect to the first, it is notable that, although in general the variables analyzed coincide with the previous broad and multidisciplinary theory on the success factors of R & D activities, what the authors provide is empirical evidence of the most important factors and variables for effectiveness in R & D; the authors emphasize that the results of the sample analyzed indicate that the most important factor is the close integration of the R & D activities with the corporate strategy, followed by the proper planning of these activities, and the achievement of financial results for the company. With respect to the relationship of the scale with other variables, the authors have found positive and significant relationships between the effectiveness in R & D and the following financial variables: net turnover and earnings after taxes. The authors have also found positive and significant relationships between different characteristics of the company and the achievement of success in R & D activities. Thus, being a company of larger size, the existence of an R & D department, the existence of specific incentive systems for the R & D personnel, the adoption of new management techniques in the R & D department, and the patents policy of the company are all factors that have a positive influence.

There are three main limitations of the study: the size of the sample; the decision to use a very particular highly innovatory sector, the pharmaceutical industry; and conducting the study in only one specific country, Spain. The results should be interpreted taking into account these limitations. Another limitation is the absence of previously validated scales. This meant that the authors were unable to do any comparative analyses.

The authors have contributed by summarizing and testing the existing theories on the factors of success in R & D. This should give R & D managers a more comprehensive and useful picture of the variables that have been considered more important, and should enable them to choose from among the range of variables proposed those that may be considered most relevant for inclusion in their own balanced scorecard. More generally, the results should help them in the management of their activity. For researchers the authors make available an already validated scale with which to work in various different samples and settings.

The originality of the work resides in two aspects. First, a very wide set of variables proposed in the literature is analyzed, with the object of establishing the relationships and the ranking of these variables, which would not be clear if the variables were analyzed in isolation. Second, there is originality in the methodology employed for measuring the result of activities with a high level of uncertainty and risk, specifically R & D activities in the highly innovative companies of the pharmaceutical industry. It is original because, to date, the scale has only been validated theoretically – there is no work in the literature validating it empirically.

Addresses the standardization of the measurements and the labels for concepts commonly used in the study of work organizations. As a reference handbook and research tool, seeks to improve measurement in the study of work organizations and to facilitate the teaching of introductory courses in this subject. Focuses solely on work organizations, that is, social systems in which members work for money. Defines measurement and distinguishes four levels: nominal, ordinal, interval and ratio. Selects specific measures on the basis of quality, diversity, simplicity and availability and evaluates each measure for its validity and reliability. Employs a set of 38 concepts ‐ ranging from “absenteeism” to “turnover” as the handbook’s frame of reference. Concludes by reviewing organizational measurement over the past 30 years and recommending future measurement reseach.

This paper aims to critically comment Rossiter’s “How to use C-OAR-SE to design optimal standard measures” in the current issue of EJM and provides a broader perspective on Rossiter’s C-OAR-SE framework and measurement practice in marketing in general.

The paper is conceptual, based on interpretation of measurement theory.

The paper shows that, at best, Rossiter’s mathematical dismissal of convergent validity applies to the completely hypothetical (and highly unlikely) situation where a perfect measure without any error would be available. Further considerations cast serious doubt on the appropriateness of Rossiter’s concrete object, dual subattribute-based single item measures. Being immunized against any piece of empirical evidence, C-OAR-SE cannot be considered a scientific theory and is bound to perpetuate, if not aggravate, the fundamental flaws in current measurement practice. While C-OAR-SE indeed helps generate more content valid instruments, the procedure offers no insights as to whether these instruments work properly to be used in research and practice.

This paper concludes that great caution needs to be exercised before adapting measurement instruments based on the C-OAR-SE procedure, and statistical evidence remains essential for validity assessment.

This paper identifies several serious conceptual and operational problems in Rossiter’s C-OAR-SE procedure and discusses how to align measurement in the social sciences to be compatible with the definition of measurement in the physical sciences.

This paper aims to test whether a structured application of persuasion principles might help improve advertising decisions. Evidence-based principles are currently used to improve decisions in other complex situations, such as those faced in engineering and medicine.

Scores were calculated from the ratings of 17 self-trained novices who rated 96 matched pairs of print advertisements for adherence to evidence-based persuasion principles. Predictions from traditional methods – 10,809 unaided judgments from novices and 2,764 judgments from people with some expertise in advertising and 288 copy-testing predictions – provided benchmarks.

A higher adherence-to-principles-score correctly predicted the more effective advertisement for 75 per cent of the pairs. Copy testing was correct for 59 per cent, and expert judgment was correct for 55 per cent. Guessing would provide 50 per cent accurate predictions. Combining judgmental predictions led to substantial improvements in accuracy.

Advertisements for high-involvement utilitarian products were tested on the assumption that persuasion principles would be more effective for such products. The measure of effectiveness that was available –day-after-recall – is a proxy for persuasion or behavioral measures.

Pretesting advertisements by assessing adherence to evidence-based persuasion principles in a structured way helps in deciding which advertisements would be best to run. That procedure also identifies how to make an advertisement more effective.

This is the first study in marketing, and in advertising specifically, to test the predictive validity of evidence-based principles. In addition, the study provides the first test of the predictive validity of the index method for a marketing problem.

Since its initial development, numerous mental health literacy (MHL) definitions and associated measures have been created which have yet to be adequately evaluated. This paper aims to evaluate the psychometric properties of global MHL measures with the aim of identifying the most valid, reliable, responsive and interpretable measure.

A systematic review was conducted of studies that evaluated global MHL measures against at least one of the COnsensus-based Standards for the selection of health Measurement INstruments (COSMIN) taxonomy properties; validity, reliability, responsivity or interpretability.

In total, 13 studies were identified which examined the psychometric properties of 7 MHL measures. Two of these seven measures were vignette format and the remaining five measures were questionnaires. The mental health promoting knowledge-10 and the multicomponent mental health literacy measure were the most psychometrically robust global MHL measures as they had the most psychometric properties rated as adequate. Both were shown to have adequate structural validity, internal consistency and construct validity. The two vignette measures, the MHL tool for the workplace and the vignette MHL measure, were both shown to only have adequate evidence for construct validity.

The current study is the first to systematically review research that evaluated the psychometric properties of global measures of MHL.

The purpose of this paper is to develop and validate a scale to reliably capture the variance of perceived risk towards purchasing on the internet at the product level.

A two‐dimensional scale to measure internet Product Purchase Risk (IPPR) was developed and tested in three studies. In the first study a preliminary pool of items was generated with evidence of content validity. In the second study the IPPR scale was purified using principal axis factor analysis. In the third study evidence of criterion‐related, known‐group, nomological, and discriminant validity was demonstrated.

The IPPR scale was found to successfully capture the variance of evaluation judgement and internet security risk. IPPR was also found to have a quadratic relationship with experience purchasing a product category from the internet.

Although strong evidence to suggest construct validity was demonstrated, it is understood that efforts to establish the validity of new measures should be ongoing. Specifically, although the IPPR scale was shown to produce reliable‐measurements for seven different products, the measurement of IPPR across more product categories would strengthen evidence of generalizability.

Given that risk is strongly dependent on the type of product, the procedure to develop the IPPR scale demonstrates the importance of measuring risk at the product category level.

This is the first study to develop and rigorously validate a multi‐item measure of purchase risk at the product level. Given the importance of risk in understanding online consumer behaviour, the IPPR scale will be useful for future studies in this domain, especially for comparing findings.

We develop and validate measurement instruments for the business partner, watchdog, and scorekeeper roles of controllers. This study addresses calls to enhance the quality of survey research in management accounting by devoting more attention to scale development and especially to construct validity. By focusing on the activity sets of the controllers’ roles, we provide a theoretically and empirically grounded picture of their current roles. The measurement instruments presented in this study enable systematic research progress on controller roles, their relationships, antecedents, and performance outcomes.

The purpose of this paper is to show how bibliometrics would benefit from a stronger programme of construct validity.

The value of the construct validity concept is demonstrated by applying this approach to the evaluation of the h-index, a widely used metric.

The paper demonstrates that the h-index comprehensively fails any test of construct validity. In simple terms, the metric does not measure what it purports to measure. This conclusion suggests that the current popularity of the h-index as a topic for bibliometric research represents wasted effort, which might have been avoided if researchers had adopted the approach suggested in this paper.

This study is based on the analysis of a single bibliometric concept.

The conclusion that the h-index fails any test in terms of construct validity implies that the widespread use of this metric within the higher education sector as a management tool represents poor practice, and almost certainly results in the misallocation of resources.

This paper suggests that the current enthusiasm for the h-index within the higher education sector is misplaced. The implication is that universities, grant funding bodies and faculty administrators should abandon the use of the h-index as a management tool. Such a change would have a significant effect on current hiring, promotion and tenure practices within the sector, as well as current attitudes towards the measurement of academic performance.

The originality of the paper lies in the systematic application of the concept of construct validity to bibliometric enquiry.