Three Methods, Four Different Outcomes! How to Test the Reliability and Validity of a Graduate Survey Competence Scale

Abstract

This chapter presents and discusses various steps to ensure empirical reliability and theoretical validity in the construction of competence scales in graduate surveys. The development of a scale to assess demands of the teacher profession and related abilities in graduates for a German tracer study project serves as an example. Confirmatory factor analysis (CFA), principal component analysis (PCA) and Cronbach’s coefficient alpha are employed to test the reliability of the scale. Differing results illustrate how the method applied influences decisions in the process of developing a scale. Our findings show that multidimensionality can only be tested appropriately by CFA; PCA renders no feasible or similar results to CFA depending on the predetermination of the number of factors; Cronbach’s alpha produces misleading results as the prerequisite assumption of unidimensionality is violated by the data.

Citation

Landmann, M., Kmiotek-Meier, E., Lachmann, D. and Lorenz, J. (2015), "Three Methods, Four Different Outcomes! How to Test the Reliability and Validity of a Graduate Survey Competence Scale", Theory and Method in Higher Education Research (Theory and Method in Higher Education Research, Vol. 1), Emerald Group Publishing Limited, Leeds, pp. 119-139. https://doi.org/10.1108/S2056-375220150000001006