Holistic versus decomposed ratings of general dimensions of work activity

Abstract

Purpose

The purpose of this paper is to examine the accuracy of the US Department of Labor's Occupational Information Network (O*NET), which, replacing the Dictionary of Occupational Titles, analyzes jobs via a hierarchical taxonomy of work in which all task‐level activities are summarized into a 42‐construct taxonomy of first‐order generalized work activities (GWAs).

Design/methodology/approach

This study examined the degree of convergence between ratings made using the holistic‐judgment process in the O*NET (which directly rates GWAs using single‐item scales) vs traditional decomposed‐judgment methods which statistically combine ratings of multiple activity items for each GWA.

Findings

Analysis of holistic O*NET general work activity ratings with decomposed common‐metric questionnaire (CMQ) ratings revealed poor convergence between holistic vs decomposed methods, low interrater agreement and a tendency for incumbents to rate higher than job analysts.

Practical implications

It is believed that these results raise significant questions regarding the O*NET's plan to rely on unverified holistic ratings obtained from relatively small, volunteer samples of job incumbents to maintain its database over time. There is a clear risk of producing a database of uncertain quality and comparability with the existing analyst ratings. It is concluded that the criteria of quality, accuracy and verifiability should be paramount in efforts to develop a national database of occupational information.

Originality/value

This study is the only empirical analysis of the degree of convergence between ratings made using the holistic‐judgment O*NET and a traditional decomposed‐judgment job analysis. Because job analysis forms the foundation for many human resources functions, effectively setting the standards that drive recruiting efforts, establishing the criteria that are used in hiring, promoting, evaluating, and equitably compensating employees, and forming the basis for many employee training programs, it is absolutely essential that any data source utilized for these purposes should be both accurate and verifiable. This study not only furthers efforts to tests the validity of the O*NET, it also provides empirical evidence of its potential shortcomings.

Citation

Gibson, S.G., Harvey, R.J. and Harris, M.L. (2007), "Holistic versus decomposed ratings of general dimensions of work activity", Management Research News, Vol. 30 No. 10, pp. 724-734. https://doi.org/10.1108/01409170710823449