Using Evaluation in Training and Development

Citation

Odini, C. (2000), "Using Evaluation in Training and Development", International Journal of Manpower, Vol. 21 No. 6, pp. 511-515. https://doi.org/10.1108/ijm.2000.21.6.511.1

The book attempts to provide comprehensive information and material about the validation and evaluation of training and development programmes. The author observes that although Investors in People insists on the inclusion of evaluation as an essential part of the learning processes, it is practised very little. Rae asserts that without evaluation nobody really knows how effective any form of training might be. As a result, money may be spent unnecessarily.

The book consists of 14 chapters, and is structured to allow trainers to develop evaluation models that ideally are suited to their individual needs. The first chapter sets the scene by highlighting the macro and micro levels of evaluation and the criteria for evaluation.

Evaluation is considered as a broad concept which includes the validation of the learning event and looks particularly at issues concerned with the application of the learning in the workplace, its longer term implementation and the cost‐ and value‐effectiveness of the training and development provided. The chapter also considers the principal reasons given by many trainers for not evaluating, the benefits of evaluating and the consequences of not evaluating.

A practical evaluation model is presented in Chapter 2. The model is, in my view, superior to most models of evaluation in the sense that it not only considers four or so items commonly associated with most models, but also describes the 14 logical and practical steps necessary to achieve an effective evaluation. The author expounds on his evaluation model by pointing out that the learning processes include the pre‐ and post‐course briefing sessions between the line manager and the learner.

All members of the organization have a role to play and every encouragement must be given to them to do so. The ideal evaluation approach, according to Rae, uses a range of role holders which he describes as the training quintet – senior managers, line managers, training manager, trainer and learner. For an effective approach all members of the quintet must play their full part in all aspects of training.

Rae observes that the basic principle behind any evaluation design is that the process must seek to confirm whether or not the training and learning objectives have been achieved, and whether they have subsequently been implemented in an effective manner. His overall requirement includes the validation of the actual training process, validation being seen as the assessment or investigation of the training and development process used to achieve learning and change.

The author outlines nine stages of test‐design, followed by specific guidelines for the design and construction of particular tests. He recommends the use of assessment observation‐activity and behavior analysis as useful methods of assessing individuals performing a task.

Rae suggests that if we need to observe a group of people to assess the level of their activity at a particular point in time, the simplest approach is known as simple contribution scoring. A slightly more complex approach is the directional sociogram, which, as its name implies, is concerned with the direction of discussion. The scoring in this approach can be linked with simple contribution scoring, and other more sophisticated activities can also be included. The author acknowledges that such observations are subjective but asserts that they are, nonetheless, valuable.

Rae observes that there are more formal and objective approaches that can, and should, be used during programmes, particularly those lasting more than a day. The more frequently used of these will include activity analysis, behavior analysis, practical tests and demonstrations, diurnal reviews or audits, spot checks, session reviews and logbook reviews.

The author cautions that there are certain dangers in introducing interim evaluation measures into the evaluation process, particularly when the learners are aware of the evaluation and are contributing to it directly. The purpose of interim evaluation is to assess how well the training is achieving its objectives and the learners are learning. Rae advises that if the interim evaluation shows that all is not progressing as it should, something must be done about this. This involves the use of time, and consequently programme resources must allow for this.

The book addresses the modes of evaluation that can be applied at the end of training and development in view of the fact that the purpose of training and development is to produce change in knowledge, skills and attitudes in the individuals. Such modes may include the use of knowledge tests and practical demonstrations, observation of general skills, observation of behavioral learning and the use of reactionaries and questionnaires.

An interesting discussion on whether the trainer or facilitator should be directly assessed by the learners is included in the book. The author notes that objective evaluation of the trainer is more difficult than evaluation of a programme, especially since the feelings of the learners may be biased for or against the trainer in many ways and for many reasons, which they may not identify themselves. Moreover, learners are unlikely to be qualified to comment on the trainers’ skills and their views may be inappropriate if they do not understand the techniques used to help them to learn.

Rae observes that the evaluation of self‐instruction (or open/distance learning or programmed instruction) is very similar to that for direct training course types of programmes. The principal difference is that for most if not all of the time that the learner is following the programme, there is no contact between him/her and a trainer.

I agree with the author’s view that too often evaluation does not extend beyond the end of programme validation, and that all that is assessed is the satisfaction of the training programme objectives and the immediate objectives of the learners. This is not complete evaluation and certainly does not lead to an assessment of the value effectiveness of the learning. Post‐programme evaluation is therefore necessary. Rae says that there is need for post‐programme debriefing, medium‐term evaluation and longer‐term evaluation

The book also addresses the issue of cost‐effectiveness and the author correctly points out that the costing of training programmes consists of in‐puts and out‐puts. According to the author, cost inputs consider one side of the financial equation, the cost of the programme and whether this has been the most economical and effective approach.

The author stresses the importance of analyzing and reporting on evaluation data. He suggests a variety of analysis methods designed to cope with evaluation data, ranging from the simplest to a complex approach involving the manipulation of hundreds (if not thousands) of bits of data. According to the author, these approaches include simple examination, test‐marking, text comparison, data and text comparison and extended data comparison

The last chapter of the book recognizes the fact that there is bound to be considerable variation in the extent to which evaluation is performed due to a number of factors ranging from lack of interest in the process to a lack of resources and time. A number of specific approaches are described starting from the simplest through an increasing complexity of methods, to full evaluation process. The author stresses repeatedly that evaluation is not for the trainer alone to carry out but is the province of the training quintet.

The value of this book is likely to be as a support handbook for trainers who are interested in using evaluation in training and development. Its straightforward style and examples, which are practical in tone yet firmly based on sound theoretical foundations, should reassure anyone who is afraid of using evaluation in training and development.

The book’s 243 pages are well worth the price. It is packed with information on the design of the evaluation process and advice on when and how to use particular types of test. The author also critically assesses the complete range of approaches to evaluation that are available. The book’s comprehensive and easy to use index is particularly useful for those who wish to dip into the book rather than read it from cover to cover. It is indeed a key tool for anyone contemplating the use of evaluation in training and development.