To read this content please select one of the options below:

Training and scoring accuracy of organisational self‐assessments

Garry D. Coleman (Virginia Tech, Blacksburg, Virginia, USA)
C. Patrick Koelling (Virginia Tech, Blacksburg, Virginia, USA)
E. Scott Geller (Virginia Tech, Blacksburg, Virginia, USA)

International Journal of Quality & Reliability Management

ISSN: 0265-671X

Article publication date: 1 July 2001

753

Abstract

This paper addresses the problem of using accuracy index values based on the squared difference between participant scores and true scores, the D2 index, at the practical level. It clarifies ambiguity existing in the literature regarding the use of these index values to evaluate the scoring accuracy of human raters (evaluators). The paper critically investigates the effect of frame‐of‐reference (FOR) training on improving the accuracy of third‐party evaluators’ scores for organisations, such as those going through the Malcolm Baldrige National Quality Award (MBNQA) self‐assessment exercise. It discusses a case study where 90 individual participants took part. The scores of these participants were recorded before training was given to them (no training) and after receiving FOR training. The study showed that providing FOR training has an effect on improving the elevation accuracy index (p < 0.05) in five of the seven categories used in this exercise. An observed leniency effect was also reduced. However, no improvement in the DA was observed. Thus, the evaluators’ ability to assign an accurate overall score was improved, while the ability to discriminate between relative strengths and weaknesses did not show improvement. This implies evaluator training, particularly for heterogeneous pools of volunteers like those of corporate and state and local quality awards, should include more content on the performance dimensions.

Keywords

Citation

Coleman, G.D., Patrick Koelling, C. and Geller, E.S. (2001), "Training and scoring accuracy of organisational self‐assessments", International Journal of Quality & Reliability Management, Vol. 18 No. 5, pp. 512-527. https://doi.org/10.1108/02656710110392827

Publisher

:

MCB UP Ltd

Copyright © 2001, MCB UP Limited

Related articles