Evaluating the quality of the learning outcome in healthcare sector: The Expero4care Model

Abstract

Purpose

This paper aims to present the Expero4care model. Considering the growing need for a training evaluation model that does not simply fix processes, the Expero4care model represents the first attempt of a “quality model” dedicated to the learning outcomes of healthcare trainings.

Design/Methodology/Approach

Created as development of the Expero model (Cervai et al., 2013), the Expero4care model has been tailored for workplace learning in the healthcare sector and tested in six organizations across Europe. The model has been validated through the review of an international team of experts and its approval as QualiCert standard.

Findings

Expero4care allows the evaluation of the quality of learning outcomes focusing on competences, impact in the workplace, transferability, participation and credits. The evaluation process involves different categories of stakeholders (learners, trainers, colleagues, managers, internal or external bodies that can benefit the training’s results, i.e. final users of the service, such as patients and citizens), and it is based on a systematic data collection and comparison among expectations and perceptions. The implementation of the Expero4care model gives the opportunity to start a continuous improvement process of the trainings in the healthcare service.

Research limitations/implications

Expero4care has been tested in both university courses and organizational trainings dedicated to professionals in the healthcare sector. The initial sample is not wide enough to cover all the countries and the types of trainings, so a larger implementation is needed to validate its appropriateness.

Social Implication

Expero4care is the first model created specifically for organizations providing training in the healthcare sector. The implementation of the Expero4care model – adaptable to different kind of organizations and trainings – means that it is possible to highlight the value of the training considering its impact on the workplace and on the citizens.

Originality/value

As the most commonly used tools to assess the quality of trainings consist of questionnaires submitted to participants at the end of the training and considering that quality models have not been utilized to analyse learning outcomes in healthcare, Expero4care represents the first quality model dedicated to training in healthcare service.

Citation

Cervai, S. and Polo, F. (2015), "Evaluating the quality of the learning outcome in healthcare sector: The Expero4care Model", Journal of Workplace Learning, Vol. 27 No. 8, pp. 611-626. https://doi.org/10.1108/JWL-09-2015-0063

Introduction

The professional development of healthcare employees is primarily based on formal training. In many countries, doctors, nurses, social workers and healthcare technicians are required to acquire specific credits throughout their career. Although the regulation of Continuing Medical Education (CME)[1] varies around the world, with vast differences in regulatory qualifications between the USA and Europe as well as other countries, such as those in Asia and the Middle East, it is widely considered the most successful strategy to guarantee, facilitate and support the professional development of healthcare employees.

One of the biggest issues, both in the literature and in practice, is how to evaluate the value of training in the workplace and, in particular, for the healthcare sector, and how to assess it in a systematic and effective way, focusing on the learning outcome.

Following the results of two European projects that aimed to create a “quality model” for assessing the quality of learning outcomes in vocational trainings and schools (Cervai et al., 2013), a new model called Expero4care is presented in its description, testing phase and results.

Theoretical background

With the aim of creating a quality model dedicated to assess the learning outcome in the healthcare sector, four main areas of literature have been explored: adult learning, quality models, healthcare education and organizational culture.

The features that an evaluation training model should have to be applicable and effective have been summarized by Griffin (2012). After an extensive revision of the literature and considering the strengths and limitations of the most common models (i.e. Kirkpatrick and Holton,), Griffin focused on the following features:

• to be robust and applicable;

• based on a stakeholder approach;

• a wide system of data collection and mix method;

• resource sensitivity; and

• time horizon.

It is therefore essential to consider all the different scenarios in which the model should be applied. Organizations themselves provide different types of trainings – long and short, with frontal lessons or “on the job”, conferences, theoretical or practical contents, online or in person – in various fields. Moreover, organizations themselves assign a set of different values to training strategies (i.e. individual or team-oriented), and the focus on training can vary greatly from organization to organization, depending on, among other things, national laws, dimension and formal mission. To be effective, the model needs to be flexible and its reliability/solidity needs to be tested in different scenarios.

The importance of a stakeholder-oriented approach has been well documented in the organizational literature and, more recently, in the Adult Education field, where it has been acquiring specific relevance, especially as learning is no longer considered “an end in itself” as an individual matter, but instead refers to the whole organization with a direct or indirect impact on the service.

The stakeholder approach also exemplifies a strategy that can be engaged to disprove the myth of finding an objective measure of the learning outcome. By taking into consideration the opinion of those stakeholders who are acquainted with the learning outcome of a training, it is possible to observe the learning outcome from different points of view. In particular, it is important to consider the stakeholders’ perspectives, bearing in mind the information they possess and their interest in contributing to the assessment.

Stakeholders’ perceptions constitute a valuable data set to analyse the quality of the learning outcome where a planned strategy of data collection, both qualitative and quantitative, is recommended to obtain high-quality data, stimulating a mix-method approach (Cortini, 2014).

To be implemented, a quality model needs to be sustainable. It is necessary to consider that, although the implementation of a model needs dedicated resources, it should not cost more than the process itself. The sustainability of the model is also related to the appreciation with the results it generates, considering that it should be able to show an output not previously known and which is useful for the organization.

Finally, a good evaluation model should aim for a medium- to long-time horizon. Learning is not a short-term process and, even when the training is brief, its results can be mainly appreciated in a medium to long period of time.

In the wider literature about training/learning evaluation, scholars have proposed as a “model” both “micro models” and “systemic models”. The first are dedicated to describing the variables that can influence the learning process, i.e. the motivation to learn (Noe, 1986), learning and retention (Baldwin and Ford, 1988) and obstacles to learning (Holton et al., 2000); while the second aims to describe the whole learning process in a systematic way: the CIRO (context, input, reaction, output) by Warr et al. (1970), the four Krikpatrick’s levels (1976) and the CIPP (context, input, process, product) by Stufflebeam, 1983, until the flawed four-level evaluation model (Holton, 1996) that connects learning, individual performance and organizational performance.

In this paper, a different approach is proposed, based on total quality management (TQM), in which the model is considered a map able to analyze criteria in a systematic way, to highlight strengths and weaknesses that can be used to improve the process. Indeed, quality models are devoted to presenting the evaluation as a process dedicated to the continuous improvement.

Among the most well-known quality models (EFQM, CAF) and ISO 9,001:2008 requirements, the training activity can be considered one of the human resource management processes dedicated to the professional development of employees. Consequently, in implementing TQM, the organization is required to indicate how it ought to monitor and improve the “training process”, and, as a result, most organizations have started to use post-training surveys to analyse levels of satisfaction with, among other things, trainers, setting, materials and timetables to collect information to improve further training. A significant limitation of this strategy is that the training is evaluated on the basis of the process itself, not on its results.

In the healthcare sector, there is a convention of considering the training process a fundamental factor required to guarantee the professional development of employees with a direct impact on the service. In many countries, healthcare employees are included in educational programs (i.e. CME) that foster the needs of professionals to be constantly updated in their job/profession. Consequentially, comprehensive literature about medical/nurses competences and healthcare professional development provides direct insight into how training has been studied in different healthcare professions. Among these studies, Calhoun et al. (2002) reviewed several taxonomies to classify and assess competences in the healthcare sector, citing the KSAV (knowledge, skills, attitudes and values) model as particularly suitable for healthcare management education. In line with the traditional classification (KSA, KSC), the extension to values has been proposed by Gagnè (1977), Stephenson and Weil (1992) and Barnett (1994), and has again been offered in the Expero4care model as a suitable classification for the deployment of competencies inside the set of indicators recommended for the evaluation.

Finally, as the model is dedicated to the training of employees in healthcare organizations, the influence of the organizational dynamics cannot be neglected. For this reason, organizational culture and climate have been identified as key dimensions impacting organizational learning. In particular, beginning with the research of Marsick and Watkins (2003), the authors highlight the importance of organizational, team and individual learning as dimensions to be considered to analyse the learning outcome of a training. Training climate has been defined (Tews and Tracey, 2008) as a short-time variable, instrumental in preparing individuals for formal development activities and in achieving desired learning objectives, constituted by work-related factors that may influence training success and failure and the effectiveness of formal and informal training activities. Organizational culture is, instead, a long-term dimension, quite stable and not easy to modify (Schein, 1984). Commonly, in the wider literature about organizational culture, training is often cited as a factor that might influence culture (Ashkanasy et al., 2000); however, at the same time, similar attention has not been paid to how organizational culture impacts training effectiveness (Benevene and Cortini, 2010). Trying to find this missing link, the Expero4care model is based on a “training culture” dimension, surveyed through a questionnaire dedicated to monitor meaning and values related to training in the organization.

Expero4care model

The Expero4care model represents the further development of previous research (Pecar et al., 2009) that has contributed to create the first version of the Expero model. Addressed to vocational trainings and recognized as a QualiCert standard, the Expero model consists of a set of guidelines and tools designed to assess the learning outcomes in schools and vocational education centres.

The Expero4care model is a tailored version of Expero which is dedicated to training programs for employees of healthcare organizations. This means that the focus is on adult learners employed in organizations (i.e. hospitals) where the mission is not education itself and where training is an internal process, mainly devoted to the continuous professional development of healthcare employees.

Following Griffin’s suggestions, the model is based on a multi-stakeholder approach, developing a systematic data collection among the main stakeholders of the training and promoting a long-term horizon. As with any quality model, it aims to identify strengths and weak points to develop an effective continuous improvement process, where the process to improve is the “training process” and the dimensions under evaluation refer to learning outcomes.

The multi-stakeholder approach provides an overall picture of the forces that may impact the learning results and on the learners’ workplace (Guerci and Vinante, 2011). Following service quality indications (Cronin and Taylor, 1992; Abdullah, 2006), the quality level derives from the comparison between expectations and perceptions. This means that data are collected in two different timeframes, before and after the training, while also considering a long-time horizon.

The learning outcome is assessed through the following criteria, or dimensions:

• training culture (TC);

• quality of competences (QC);

• quality of results (QR);

• satisfaction with results (SR); and

• training processes (TP).

Each dimension refers to different stakeholders to collect data from those who have information. Data are collected using a mix-method approach, including both qualitative and quantitative data.

The stakeholders have been classified into five categories:

1. Leadership: It is represented by the manager of the organization, and in the Expero4care implementation is the one who decides which strategic trainings to monitor.

2. Decision maker: He/she the one who requires the strategic training, usually the one who identifies the need of specific competencies and asks for the realization of a specific training.

3. Internal stakeholders: They are the ones who are directly involved in the realization of the training (i.e. teachers, tutors and administrative staff of the training office).

4. Learners: They are those who participate in the training. They are considered in a specific category because they are simultaneously actors and beneficiaries of the learning outcome.

5. External stakeholders: They are those who benefit from the learning outcome, although they are not directly involved in the training and are divided into:

   • Workplace: People belonging to the structure where learners work (supervisors, managers, colleagues and employees).

   • Professional network: Professional advisers, healthcare associations and services or people with whom the learner is related during his/her work activity.

   • System: Healthcare system and citizenship and monitoring of significant data in which the training should impact.

The graphical representation (Figure 1) shows the whole model and consists of two areas (“Should” and “Is”) and five rows, one for each stakeholders’ category. The boxes represent the five dimensions in their articulation among time to survey (“Should” and “Is”) and stakeholders.

The two areas (“Should” and “Is”) indicate the different timeframes in which data need to be gathered: before and after the training. In the “Is” area, two dimensions (QC and SR) are surveyed directly at the end of the training, while QR is surveyed after three to six months to appreciate possible changes in the workplace.

The Expero4care model aims to evaluate the learning outcome, which has been deployed in six indicators:

1. Competencies: It is Identified using the KSAV model, deployed for each training aim:

   • Knowledge: It is a theoretical knowledge acquirable through the training;

   • Skills: These are the abilities obtainable through the training;

   • Attitudes and behaviours: It is expected in the learner performance; and

   • Values: These are new or reinforced meanings to apply on the job.

2. Transferability at workplace: It implies the possibility to transfer to the colleagues what was learned during the training;

3. Applicability at workplace: Possibility to practice what was learned on the job;

4. Impact: Effects of the training on the organizational outcomes (i.e. improvement of the quality of the service and decreasing of the costs);

5. Participation: In terms of the number of participants and the level of attention of the learners; and

6. Credits: Appropriateness of the number of credits assigned. It refers to the credit system, actually used in a lot of countries (including the USA and those in the European Union) as a formal acknowledgement/numerical indicators of the attended training. It could be specific for the medical profession or related to university system.

Each dimension refers to a set of tools (semi-structured interviews, questionnaires, indicators/outcomes) created and tested to collect data from each stakeholder to survey the above-mentioned indicators.

Implementation of the Expero4care model

The core of the Expero4care model implementation is the Training Evaluation Board (TEB): a team of people – experts in training – who manages the whole evaluation process and the data collection of the monitored trainings. The main tasks of the TEB are: supervising the application of the model, rating the gaps between expectations and perceptions and proposing improvement and corrective actions to the identified gaps.

The first dimension is TC and can be defined as the meanings and values attributed to the training activities in a specific organization. It refers to the formal learning in an organization and focuses on stable and durable features related to the training. Data are collected through a questionnaire, pre-tested in six organizations, constituted by 26 items divided into three dimensions (individual, team and organizational levels) and some additional questions to classify employees.

The leadership contributes to the process of defining the vision of the organization about training and learning, by selecting in the TC questionnaire the items useful for describing the TC. These data will be compared with the data collected from the employees of the organization.

Through the TC dimension, it is possible to explore meanings and values attributed to the training in an organization adopting an organizational cultural perspective. It allows comparisons between the leaderships’ and the employees’ points of view to define an organizational profile in terms of values and meanings attributed to the training. During the evaluation process of this dimension, the TEB needs to focus on the possible gaps in:

• significant distances between the leaders’ and employees’ profiles; and

• the limited homogeneity of the employees’ sample.

In the first case, when the values declared by leaders are significantly different from the values expressed by employees, quality can be improved by identifying effective strategies to converge. In the second case, data show that there is not a strong enough culture regarding training and its meaning and values are very broad. A strategy that enables sharing a common intention for various trainings while considering the profile of individual leaders can also contribute to reinforcing the organizational culture and avoiding misunderstandings in the training policies.

In implementing the model, a preliminary step consists of identifying the strategic trainings to monitor. However, the model has not been created for a wider and potentially generic evaluation, rather it has been designed to facilitate an in-depth analysis of the core trainings so that just the strategic training within an organization will need in-depth monitoring.

The strategic value of a training is assigned by the leadership (such as the chief executive officer [CEO], manager or the president) who ideally should know the appropriate professional development strategies of the personnel.

Once the leadership indicates the trainings that need to be monitored, the evaluation process can take place according to the following stages initially consisting of activities that must be managed before training commences (Should phase):

• weighing stakeholders and indicators;

• defining stakeholder’s sample;

• collecting data about external stakeholders’ expectations (QR); and

• sharing the training aims and the deployment of competences (QC).

As the model can be tailored to different organizations and trainings, the first step consists of choosing the stakeholders’ categories and indicators, weighing them and considering their importance, i.e. the higher the score, the higher the importance attributed to that stakeholder/indicator. Alternatively, when the assigned weight is zero, the indicator/category will automatically disappear from the survey.

Stakeholders involved in the training are people or bodies considered important enough by the organization to appreciate the learning outcomes and are also interested in the quality of the learning processes. External individuals/organizations are involved to analyse the impact of specific training on a workplace, on patients/citizenship and on the whole healthcare system. Once they have been assessed, the TEB and the decision maker identify the persons to be interviewed and the indicators of outcome to be monitored.

QR is composed of four boxes, two in the “Should” category and two in the “Is” category. Data are collected during two different timeframes involving the decision maker and the external stakeholders. Before the training starts, the decision maker and the selected external stakeholders are interviewed to understand their expectations about the learning outcomes of the specific training. The indicators required to examine the outcomes need to be monitored before the training starts. These results are compared with a second data collection which takes place three to six months after the end of the training, during which the same stakeholders are interviewed again and asked to describe their perceptions about what the learners have actually learned and subsequently have applied and transferred in the workplace. The outcomes’ indicators are also monitored again to adjust any changes with previous levels.

In evaluating data related to the QR dimension, TEB needs to focus on the possible gaps between expectations and perceptions that are signals of a low quality and a potential source of criticism in learning outcomes. When a stakeholder has higher or different expectations of the evaluated training or learning outcome, the quality level of the training will not be satisfactory for the subject. This dissatisfaction can derive from a lack in the communication process (i.e. the stakeholder was not correctly informed about what the training provides), a lack in the training process (the training was not effective), a lack in the learning process (trainees do not acquire the expected competences) or in the organizational process (the workplace does not permit to apply what learned). Any of these gaps can be monitored through the data set, and the TEB should highlight the most critical to plan possible solutions.

Whereas the QR is mainly dedicated to the external stakeholders and characterized by the comparison of expectations and perceptions, the QC is mainly oriented to learners and involves trainers’ evaluations. The first step consists of the formulation of the deployment of competences of each training aims by adopting the KSAV model and needs to be shared among decision makers, learners and teachers before the training starts. The sharing process can be managed in different ways and depends on the organization processes. Sometimes it is led by the decision maker and training office, and other times the learners have a more active role in their definition; what is important is to make a common decision about training aims and competences that a learner is supposed to acquire and a trainer to teach, which should be coherent with the expectations of those who decided that the training has to be managed (the decision maker).

Right at the end of the training, two surveys should be conducted to allow the comparison between learners perception of acquired competences to the trainer(s) evaluation.

Trainers are asked to declare the levels gained by the class for each unit of competence and the degree of its homogeneity in the class. Through a self-evaluation questionnaire, each learner declares for each unit of competence their initial and final levels. In addition, some specific items are dedicated to the applicability on the job and the possibility of transferring to colleagues what has been learned.

The data elaboration system included in the Expero4care database provides graphs showing (Figure 2) the position of each learner before (x-axis) and after (y-axis) the training (blue points) measuring their distance from the teacher/s position (green triangle). The three red shadows show three different thresholds in terms of achievement of the specific competence (critical, intermediate and good levels). A large distance between teacher and learners perception, or a wide dispersion of the points in the graphs, highlights the critical points that need to be analysed while considering the quality of the learning outcome.

Finally, the Expero4care model is completed by two dimensions, generally required by quality models: TP and SR.

TP dimension refers to the management of training activities from the administrative, organizational and teaching points of view. This dimension aims to describe the organizational processes related to the training on the basis of: need analysis (methodology, timing and results), annual planning, the training program, the management and communication of the training inside the organization and the dedicated resources (including training materials, setting and the financial plan of the training).

TP dimension is not under the TEB evaluation because the Expero4care model focuses on the evaluation of the learning outcomes, not on the TP. In addition, a clear definition and description of these processes allow information to be systemized so that, where an improvement needs to be planned, there is a clear statement of the related processes.

The SR dimension refers to the learners’ satisfaction with the training. It consists in the questionnaire usually submitted by the training office at the end of each training, in which learners are invited to express their perceptions of several training issues (such as setting, teachers and timing). Because it is a consolidated process in most organizations, Expero4care guidelines do not provide a specific tool, it only includes a single item dedicated to the satisfaction level with what has been learned (on a scale of 0-100).

Despite the fact that in adult education literature, many studies have demonstrated the tiny correlation between reaction and learning (Mathieu et al., 1992), it is a fact that most quality models (ISO 9000 first) consider the level of satisfaction as the main – or even the only – indicator to assess the quality of learning by the trainees. Questionnaires filled in at the end of the training remain one of the most commonly used practices to analyse the learning outcome.

In evaluating the SR dimension, the TEB is invited to summarize the results of the questionnaire, while the database automatically elaborates the answers to items related to the satisfaction levels. Reflecting on this summary, the TEB needs to underline any critical situations and evaluate how to improve them.

All the data collected are stored in the Expero4care database and summarized in an effective way to facilitate the TEB in the final evaluation. For the qualitative data (i.e. QR), the system provides an output step by step, competences by competences and stakeholder by stakeholder. To perform the QR evaluation, the TEB needs to carefully read the related interviews to evaluate the quality level of each dimension on a scale from 0 to 100 and summarize the situation in a specific box (TEB remarks). For the quantitative dimension (i.e. QC), the TEB analyses the graphs, attributing a score (0 to 100) to each one and collecting the main features observed in the TEB remarks box.

When all the TEB evaluations are registered in the system, they are re-processed considering the initial values attributed to each stakeholder’s category and to each indicator. A final table containing the TEB remarks and scores highlights the strengths and weaknesses of the quality of the learning outcomes in the monitored training. Following the Pareto principle[2], the algorithm clearly illustrates the most relevant gaps on which an improvement action is requested.

Expero4care model: testing phase

The Expero4care model has been developed through a European project (LLP-TOI) aimed to create, test and certify the model as a quality standard.

To validate the new model, the research team used a three-stage approach consisting of theoretical design of the model and tools, test in organizations and external validation.

The first step consisted of elaborating the theoretical model, starting with the literature review (see paragraph 1), discussing it with international experts, presenting it in international conferences[3] and developing the design, contents and tools to be applicable in the testing organizations.

The second step refers to the implementation of the model by the partner organizations to test the tools and the reliability of the achieved results.

To create a model flexible enough to be easily adapted to various types of trainings in the healthcare sector and conformable to different implementing organizations, the testing phase has been conducted in six organizations in five EU countries (Croatia, Finland, Italy, Spain and Sweden), monitoring 12 trainings, with different targets and learners.

Table I summarizes the testing phase. It is possible to observe testing bodies: healthcare organizations (private and public) and universities. The former are organizations where healthcare is the core mission of the company, and trainings activities constitute a strategy for the professional development of their employees, with different rules for public and private ones, and are also related to national law. The latter are universities where courses and trainings represent the core of the organization.

All the monitored trainings have been dedicated to adult learners already employed in the healthcare sector and belonging to different categories: doctors, nurses, social workers, psychologists, medical technicians (in medical laboratories and IT) and paramedics.

A various set of trainings have been included in the sample: from short ones (three hours) to longer ones (an academic semester, 28 hours); theoretical and practical; inside or outside the organization, with one or multiple teachers, homogenous or heterogeneous classes; and using different didactical approaches.

During the testing phase, some minor adjustments were made to adapt the model to different realities, languages, and cultural and organizational needs.

The third step consisted of validating the theoretical model and its tools through its recognition as a quality standard. For this scope, an international certification body has been involved (SGS international) which thoroughly analysed the model, following a strict process of review, structuring and verification of the international standards for the accreditation of the model, and, finally, recognized it as a QualiCert standard devoted to assess the learning outcome of training courses in the healthcare sector. Once approved by the SGS international committee, six organizations have been allowed to require an external audit to receive the Expero4care QualiCert certification.

Case study

To exemplify the application of the Expero4care model, a case study is presented considering one Italian public organization in the healthcare sector (Azienda per l’Assistenza Sanitara n.1 Triestina – AAS1), where seven trainings have been monitored through the Expero4care model. The case study pointed out the main issues and the relevant features of the model implementation in one of these trainings.

AAS1 is a public company, situated in the North East of Italy. The company’s mission states promotion of health, mainly by providing healthcare services in the local area, with strong connections with the local hospitals. It provides a large number of healthcare services, divided into four geographical districts and manages: daily care to medical analysis, paediatric and family caring, social work assistance, certification of public health standards places, drugs and alcohol dependence and mental health assistance. It has 1,285 employees, mostly nurses, doctors and other healthcare professionals (900) and the remaining third being administrative staff (200). The training office is composed of five employees, and it provides around 150 trainings per year.

The implementation of the Expero4care model started by creating the TEB team composed of two experts in training and one expert in quality management.

The CEO identified the strategic trainings to monitor with the model by considering their importance in the organizational development and the regional policies that underline the healthcare priorities to focus on annually. Among them, the case study focused on a practical training for physiotherapists aimed at developing and reinforcing the competences of the professionals in the treatment of lymphedema of the upper and lower limbs (Lymph-Training). The training had strategic value for the organization because it was thought to solve a critical situation in the provided services, with a direct consequence for the health of the patients: before the training, there was only one physiotherapist able to perform a specific technique for lymphedema; and due to the increasing number of patients, the waiting list was very long and caused overlapping in the standards of the organization. The solution proposed by the coordinator of the physiotherapists was to train more physiotherapists about this technique to provide them with the essential skills necessary to offer this therapy in more districts and, consequently, to reduce the waiting list. The coordinator of the physiotherapists was individuated – following Expero4care guidelines – as the decision maker of the training.

In implementing the Expero4care model, once the training has been selected, there is the need to evaluate and select stakeholders and indicators, defining the sample of people to be interviewed. Rating the importance of each stakeholder and indicator during the preliminary phase allows, in the evaluation phase, to highlight the main gaps and improvement points.

The TEB, supported by the decision maker, rated the stakeholders and learners in the order of highest importance and identified ten people to interview, most of whom were colleagues of the physiotherapists and their direct supervisors. They identified, as an indicator to monitor the outcome, the “number of patients treated in each district”, before and after the training.

Considering the six indicators of learning outcomes, they gave the higher importance rating to competences and the lower rating to participation.

The ten interviews to external stakeholders (QR) in the “Should” phase had been managed before the training and stored in the Expero4care database. The outcome indicator had been recorded in the same period.

In the following steps of the Expero4care implementation, the TEB defined the training aims and the relative units of competences (QC), using the KSAV model. They identified three units belonging to Knowledge, Skills and Attitude areas. The deployment of competences had been approved by the decision maker and presented to the learners during the first session of the training. The training was 20 hours long, divided into five sessions, from May to June 2014.

Right after the end of the training, the same list of competences had been re-presented to the learners in the form of a questionnaire, asking for each unit of competence the current and the previous level. They were also invited to answer questions about the applicability and transferability in the workplace. The trainer filled in a similar questionnaire, indicating the level of the whole class for each unit of competence and the level of homogeneity of the class in acquiring competences. The database processed all the data giving as output graphs, as shown in Figure 2, where it is possible to see the low level of homogeneity among the physiotherapists, both at the beginning and final levels. Moreover, what is more critical is the high score assigned by the trainer (100, see the green triangle in the Y-axis of Figure 1) compared with the varied self-perception of learners. In analysing these data, TEB pointed out the difficulty for the trainer to recognize the level acquired by participants, due to the fact that she mainly considered them as colleagues rather than learners. Moreover, TEB highlighted the low level declared in the theoretical unit, identifying the need to provide physiotherapists with a new training primarily focused on theoretical knowledge. Indeed, in the planning of the first training, theoretical knowledge had been considered to be already acquired by the physiotherapists, and, therefore, trainers treated it in a very synthetic way.

Six months after the end of the training, a new cycle of interviews had been submitted to the same sample of external stakeholders (QR). All the qualitative data stored in the database were carefully read by the TEB, crossing expectations and perceptions by each stakeholder’s category, for each indicator. The database supported this process by filtering the interviews and deleting personal information about interviewees (the data were submitted anonymously). The TEB individuated two main gaps, the first related to competences, the second the applicability of what was learned in the workplace. In both cases, the expectations are higher than the perceptions. From the interviews’ tracks, it is evident that colleagues and supervisors expect to see a genuine application of the treatments for the patients; moreover, they also expect to be able to open more dedicated health centres in local districts.

TEB explained these gaps by stating that the assumptions of the high level of expectations probably derived from the high satisfaction level for previous similar trainings. Indeed, the same trainer managed a similar training with another group of physiotherapists the previous year. It was assessed only by questionnaires at the end of the training that showed high scores among the learners and, therefore, created high expectations towards this new version.

In fact, in the second cycle of interviews, six months after the end of the training, stakeholders (colleagues and supervisors) admitted that learners did not appear to have acquired enough competences during the training; moreover, they were not able to apply what had been learned in autonomy.

TEB decided to consider these gaps mainly as a communication problem with stakeholders. It was evident, from their point of view, that one training session could not guarantee the immediate application of the new treatment in an autonomous way and that the level of competences needed to be increased through new trainings.

In fact, both the previous edition and the monitored training received very high scores (95 per cent) by trainees with a high level of satisfaction. On the one hand, it clearly indicates that the trainer was considered well prepared and that the trainees were satisfied about what they learned; on the other hand, it demonstrates that learners’ satisfaction is not always a reliable proxy to measure the quality of a training.

To conclude the evaluation process, TEB introduced into the database all the required data about TP, while the TC dimension was included in a more general survey for the whole organization.

When all the data and TEB evaluations have been stored, the Final Evaluation gave as output a classification of the gaps, considering the initial value assigned to indicators and stakeholders. In this new ranking, following the Pareto principle, three main gaps have been highlighted with a prior level to intervene so that the final step for the TEB was to plan improvement strategies to reduce these gaps.

Summarizing this experience, the implementation of the model highlighted several gaps related to the quality of the learning outcome: the need to acquire deeper theoretical knowledge and skills, to select trainers able to evaluate the level of acquired competences and to better inform colleagues and supervisors that a training does not guarantee an immediate solution to an organizational problem. All these gaps have been detected through the Expero4care model’s implementation, and they would have passed unnoticed if a systemic analysis of data had not been led, or by only reviewing the answers given on the questionnaires by trainees at the end of the course.

Limitations and further development

Expero4care has been designed and developed in an effort to answer the organizational need to evaluate trainings’ effectiveness. Although it is neither a return on the investment of the training nor a tool to compare and rank different training, it permits the evaluation the quality of the learning outcome. Its strengths lay in its multi-stakeholders’ approach, systemic data collection, strong orientation towards a continuous improvement strategy and on a long-term view, coherent with Griffin (2012) indications.

The model has been dedicated to trainings in healthcare sector, but it could be extended to different sectors. The first testing process demonstrates its adaptability to several countries and different types of organizations within the healthcare sector. To continue the validation process, a comparison with different training evaluation models needs to be proposed.

Furthermore, the testing organizations acknowledged, as a main value add in implementing the Expero4care model, the opportunity to discover improvement areas that would have been otherwise ignored. The recognition as a QualiCert standard gives to the Expero4care model the confirmation, by a third party, of the respect of the quality criteria of a continuous improvement process (ISO 9001:2008).

As the model has been only tested in a limited number of European organizations, a wider test phase in different national healthcare systems could help to understand its transferability and opportunities for improvement.

Figure 1.

Graphical representation of the Expero4care model

Figure 2.

Graphical view of QC results, comparing self-perception by learners and trainers evaluation for a single unit of competence

Table I.

Summary of trainings, trainee and organizations testing the Expero4care model

Notes

i.e., Kontinuierliche Berufliche Entwicklung – KBE in German; Formation médicale continue – FMC in French; and Educazione Continua in Medicina – ECM in Italy.

In the various literature about TQM tools, the Pareto principle is seen as a parameter used to draw attention to the upper level (20%) of critical points to be improved using 80% of resources. It is a way to suggest that not all the problems can be immediately solved; improvement actions need resources to become effective and it is better to start with the more serious problems.

The Expero4care model has been presented at several international conferences.

Corresponding author

Sara Cervai can be contacted at: cervai@units.it