The impact of leadership development on GP mental health commissioning

Abstract

Purpose

This paper aims to explore the learning needs of general practitioners (GPs) involved in commissioning mental health provision in England, and offer an evaluation of a leadership and commissioning skills development programme for Mental Health Commissioners.

Design/methodology/approach

Retrospective mixed method, including online mixed method survey, rating participants’ knowledge, skills, abilities, semi-structured telephone interviews and third-party questionnaires were used. Results were analysed for significant differences using the Wilcoxon Signed Ranks test. Open-ended responses and interview transcripts were analysed thematically.

Findings

Indicative results showed that participants perceived significant impacts in ability across eight key question groups evaluated. Differences were found between the perceived and observed impact in relation to technical areas covered within the programme which were perceived as the highest scoring impacts by participants.

Research limitations/implications

The indicative results show a positive impact on practice has been both perceived and observed. Findings illustrate the value of this development programme on both the personal development of GP Mental Health Commissioners and commissioning practice. Although the findings of this evaluation increase understanding in relation to an important and topical area, larger scale, prospective evaluations are required. Impact evaluations could be embedded within future programmes to encourage higher participant and third-party engagement. Future evaluations would benefit from collection and analysis of attendance data. Further research could involve patient, service user and carer perspectives on mental health commissioning.

Originality value

Results of this evaluation could inform the development of future learning programmes for mental health commissioners as part of a national approach to improve mental health provision.

Citation

Dickerson, E., Fenge, L.-A. and Rosenorn-Lanng, E. (2017), "The impact of leadership development on GP mental health commissioning", Leadership in Health Services, Vol. 30 No. 3, pp. 343-351. https://doi.org/10.1108/LHS-09-2016-0046

How this fits in

While there is much research relating to theories of leadership and leadership development (Storey et al., 2015), there is limited published research relating to the impact of leadership development. A study of 55 leadership evaluations (Russon and Reinelt, 2004) concluded that programmes often need to demonstrate immediate results for funders, resulting in the evaluation of short-term outcomes. Funding of mid- to long-term evaluations were considered to be of lower priority. Establishing effective methods to evaluate impact may be a barrier particularly in complex environments such as healthcare improvement (Carr et al., 2009). Measurements of impact can also be difficult to quantify (Dierckx de Casterlé et al., 2008). This lack of research is arguably because impact is often complex and costly to assess.

Of the published literature, some have examined enablers for embedding learning from leadership development programmes into practice. Block and Manning (2007) evaluated a leadership programme involving 92 frontline leaders working across acute and community healthcare settings. The authors concluded that in addition to a well-structured program, systemic support for leadership development from line managers and organisations is critical in ensuring leadership competencies can be embedded in practice. Others have explored the impact on skill and competency development. Dierckx de Casterlé et al. (2008) completed a single case study evaluation which examined the impact of leadership development on the interaction between leader and co-workers. The findings showed a positive impact on team interactions as a result of leadership development leading to better communications within the team; although the evaluation was not able to conclude whether this had impacted the care giving process. Purdy (2016) examined the link between programme design and leadership development, concentrating on the impact of a leadership institute on the career development of Canadian healthcare professionals. The findings identified that a mix of theoretical and experiential-based learning positively impacted leadership competency.

At the time of writing, no literature could be found which examined the development of the GP commissioner role. The GP commissioner role is a more recent innovation, and as a result the role is still developing. This presents new challenges and motivates the need for evaluating leadership and commissioning programmes to support the development of this new professional role. The indicative results of this evaluation show a positive impact on leadership and commissioning practice has been both perceived and observed. Programme characteristics contributing to impact could be likely success criteria for future programmes.

Summary

General practitioners (GPs) are required to continually update their skills to keep pace with the transformation of primary care services. In recent years, this has included the need to develop leadership and commissioning skills. The challenges of integrated care and the transformation of mental health provision, alongside the devolution of health care budgets to Clinical Commissioning Groups (CCGs), requires GPs to adapt. However, GPs and GP Practices often find it difficult to prioritise their own learning and development in these areas.

In 2012, the Strategic Clinical Network East of England established a Leadership and Commissioning Skills Development programme for Mental Health Commissioners focused on developing GP leadership and commissioning skills within the context of delivering mental health services. The rolling programme offers a series of bi-monthly workshops on identified development priorities, including leadership and technical commissioning. Participants are also able to access one-to-one mentoring and knowledge facilitation meetings based on individual need. Participation in the programme is flexible with participants choosing which aspects of the programme they access.

This paper explores an evaluation of this programme. The evaluation found improved performance specifically related to working effectively with others, supporting others to improve performance, leadership and influencing skills. It is hoped that the results of this evaluation will inform the development of future learning programmes for mental health commissioners as part of a national approach to improving mental health provision.

Introduction

The Health and Social Care Act (Department of Health, 2012) established Clinical Commissioning Groups (CCGs) giving GPs the lead for “transformation” of service provision (NHS England, 2013). Transforming and improving mental health provision is a key government target and a key requirement for CCGs. Alongside additional funding of over £1bn a year by 2020/2021 (Mental Health Taskforce, 2016), GPs are expected to deliver rapid improvements in mental health services.

In 2014, “Closing the gap: priorities for essential change” (Department of Health, 2014) set out 25 areas for urgent action including:

• better integrated physical and mental health care;

• reduced suicide rates; and

• improved access to psychological therapies.

This requires that GPs are equipped with the necessary skills to lead service innovation and commission integrated provision (Department of Health 2014; Ham and Murray 2015).

GPs are at the forefront in terms of leading changes in National Health Service (NHS) provision (Department of Health, 2010), and require skills to empower those with mental health needs as they innovate services (HM Government, 2011). Innovation includes the establishment of GP consortia and joint commissioning arrangements, requiring GPs to develop leadership capacity including an ability to learn and adapt at an organisational level (Giordano, 2011; Lester et al., 2004). Although GPs are pivotal in transforming services, they are more likely to be involved in “implementation leadership” as much of the direction concerning new models of care originates from national bodies such as NHS England (Storey et al., 2015).

Leadership which embraces innovation is key to supporting integrated services (Naylor et al., 2016). Building trust is a key aim for those leading integrated mental health provision through relational approaches built on common values and networks (Porter et al., 2013). Clinical leadership and leadership roles are an important part of GP commissioning, but this requires sufficient space to engage with wider strategic issues (Miller et al., 2015). This was reflected in the design and delivery of a leadership programme to improve the mental health commissioning skills of GPs and other lead commissioners. The Strategic Clinical Network East of England delivered workshops focused on needs assessment, system redesign, exemplar specifications and the use of levers and mechanisms to drive improvement. The evaluation set out below highlights some of the key strengths of this approach.

Methods of evaluation

The evaluation was completed during April 2015 to March 2016 using a retrospective mixed method approach adapted from a method developed by The National Centre for Post Qualifying Social Work (Keen et al., 2014). Eight key question groups were defined by applying the NHS Leadership framework (NHS Leadership Academy, 2011). A retrospective approach was taken, as evaluation was not included within the initial programme design and programme timescales did not allow for this to be completed prospectively. Data were collected at three stages. Stage 1 was an online mixed method questionnaire, which asked participants to rate their knowledge, skills and abilities prior to taking part in the programme, and at the present point in time. Open-ended questions were included to gain examples of impact on practice, barriers to applying learning and programme feedback. Semi-structured telephone interviews with participants took place two months after the questionnaires, to further explore impact on practice. All participants were asked to identify an appropriate third party (line manager or colleague) who would be able to comment on any observed changes in practice. Third parties completed retrospective online questionnaires, which included pre- and post-programme skill rating questions. Confirmation of the working relationship between third parties and participants was also collated within the questionnaire.

Sample

Participation within the programme was defined as attending one or more workshops and/or accessing other programme aspects such as facilitated meetings, one to one coaching or receiving information distributed by the programme via email. In all, 106 people were identified as programme participants, of which 36 per cent (n = 38) had attended a workshop(s). Information was not available in relation to number of people accessing other parts of the programme. All participants (n = 106) were emailed an online self-evaluation questionnaire, and asked to take part in a telephone interview.

Analysis

Questionnaire results were analysed for significant differences using IBM SPSS v20. Changes were identified using the Wilcoxon Signed Ranks test to ensure findings were robust and consistent across the sample, measuring the relative percentage increase between pre- and post-response scores. Open-ended question responses and interview transcripts were analysed thematically to add context to these results.

Results

Seventeen per cent (n = 18) completed online questionnaires; this included 42 per cent (n = 16) of those who had attended a workshop(s). Eleven email responses indicated that people felt they had not participated in the programme and felt unable to comment.

Four completed semi-structured telephone interviews (four also completed an online questionnaire) and five third parties completed online testimonies of which three were line managers and two colleagues of the participants. Fifty per cent (n = 9) of the evaluation participants categorised themselves as GP commissioners. Of the 17 respondents who answered this question, 41 per cent (n = 7) had been in a commissioning role for less than two years, 41 per cent (n = 7) worked in a commissioning role between two and eight years and 18 per cent (n = 3) for eight years or more.

Results shown represent those who attended workshops (n = 16). Table I shows the relative increase on perceived impact. All results were significant with the exception of the impact on participants self-awareness. Qualitative feedback suggests that timing of the programme in relation to length of time in role influenced the perception of programme impact on individuals practice. Those newer to the role described the learning area, “understanding healthcare systems and NHS structure” as key to personal development.

Results were segmented by role (GP and non-GP Commissioner) and length of time in role to analyse impact. Findings indicated that programme impact was less significant for non-GP commissioners, although this could be due to the small sample size. It is worth noting that qualitative feedback indicates that non-GP commissioners were not always able to attend relevant workshops either due to time or other work pressures; however, further exploration would be required to fully understand this.

Figure 1 illustrates the mean average relative percentage increase per question group. Participants perceived the highest impact in relation to the Crisis Care Concordat; understanding of healthcare systems and the structure of the NHS; and successful, safe and ethical decommissioning of services. Working with others and leadership skills and qualities were perceived as the areas least impacted by the programme.

Perceptions of least impact were attributed to oral, written, non-verbal communication skills and working within teams. These were amongst the highest pre-score ratings of all question groups, suggesting the majority of participants felt these skills and abilities were among the most developed prior to beginning the programme. The most common change in practice as a result of the programme, however, was working effectively with others, with just under half of the participants reporting this, although this could be due to social desirability bias.

Table II summarises third-party testimony results (5 of 13 questions showed a significant impact) although this could be due to small sample size. Cross-referencing third-party results with self-perceived ratings as illustrated in Table III demonstrates a validated impact score. Participants perceive their skill, knowledge and understanding have increased and changes have also been observed by line managers/colleagues representing organisational impact.

Discussion

Summary

Indicative results show a significant positive impact in five learning areas:

1. leadership;

2. influencing skills;

3. leading change through people;

4. supporting others to improve performance; and

5. working effectively in partnership with other organisations.

Results highlight differences in perceived and observed impact of the programme, including significant impacts in ability across the eight key question groups evaluated. Differences were seen in relation to technical areas covered within the programme: the Crisis Care Concordat; and commissioning and decommissioning of services, which were perceived as the highest scoring impacts. These learning areas were amongst the lowest pre-programme scores, indicating this was an area participants were least confident in prior to taking part in the programme. All participants also reported an overall increase in confidence; however, this was not reflected in third-party observations. This may be due to sample size or that these skills were not observed by third parties.

Areas reflecting the smallest increase in self-reported skills were oral, written and non-verbal communication skills and working within teams. These were the highest pre-score ratings from the question groups, suggesting that participants felt these areas were among the most developed prior to beginning the programme.

The most commonly reported gain was developing networks and sharing knowledge between colleagues; 43 per cent (n = 7) of those who responded to this question reported this. This pattern was also observed in interview feedback; all (n = 4) described developing effective working relationships and examples of knowledge sharing.

Strengths and limitations

The sample size was not statistically significant. Due to sample size and self-selection (n = 18), all results are indicative.

Response rates from participants who did not attend workshops but who were defined as part of the programme were very low (n = 2). Feedback suggests that those who did not attend a workshop did not identify themselves as part of a development programme.

Those who were very satisfied or very dissatisfied were more inclined to respond than respondents who were undecided. However, it cannot be assumed that good or bad rates are a result of this bias. Retrospective methods may impact data quality due to limitations in participant recall and bias. Pre- and post-questionnaire design may lead to social desirability bias. Future evaluations would benefit from prospective evaluation methods.

Comparison with existing literature

While there is much research relating to theories of leadership and leadership development, published research relating to the impact of leadership development programmes are limited. These early findings concern an important and topical area which highlights the need to establish a body of evidence to increase understanding of the impact of leadership development initiatives on performance and practice.

Implications for practice

The indicative results show a positive impact on practice has been both perceived and observed. Findings illustrate the value of this development programme on both the personal development of GP Mental Health Commissioners and commissioning practice.

Implications for research

Although the findings of this evaluation increase understanding in relation to an important and topical area, due to the size of the sample, results are indicative. Larger scale, prospective evaluations are required to test whether identified elements positively impact on mental health commissioning and leadership practice more generally. Impact evaluations could be embedded within future programmes from the outset, to encourage higher participant and third-party engagement. Future evaluations would benefit from collection and analysis of attendance data. Further research could involve patient, service user and carer perspectives on mental health commissioning.

Conclusion

Participants and third parties reported significant impacts in the following areas:

• leadership skills;

• influencing skills;

• leading change through people;

• supporting others to improve performance; and

• working effectively in partnership with other organisations.

This demonstrates a validated positive impact on working effectively with others, supporting others to improve performance and leadership and influencing skills. Findings suggest the programme has effectively prioritised key developmental areas for GP commissioners, creating positive practice. This indicates that the programme has positively impacted soft skills and supported participants to improve performance through leading change.

The biggest gain reported was developing working relationships and extended networks. Face-to-face workshops can have a positive impact on networking and commissioning practice. The mixed group format was considered beneficial by participants who were also involved in determining topic areas for future workshops ensuring the programme content was relevant for a group of mixed learning needs.

Contextual factors influenced the impact of the programme; six participants were not able to attend relevant workshops, with conflicting workloads described as the primary reason for this. This suggests individuals and organisations struggle to prioritise personal development.

Participants perceived the main barriers to implementing learning as time and financial constraints.

Figures

Figure 1.

East England development programme: self-perceived relative percentage increase mean average per question group

Table I.

Participant self-completion questionnaire results; pre- and post-programme ratings

Question group	Z a	Asymp. significance (2-tailed)	Relative % increase
Understanding of healthcare systems and the structure of the NHS	−2.936b	p = 0.003	42
Leadership skills and qualities	−2.805b	p = 0.005	23
Commissioning skills and knowledge	−2.807b	p = 0.005	35
Working with others	−2.668b	p = 0.008	23
Programme specific learning areas: (Crisis care concordat, decommissioning of services, Parity of esteem SMI CQUIN, Interpreting data sets)	−2.812b	p = 0.005	42

Notes:

^a

Wilcoxon Signed Ranks Test;

^b

Based on negative ranks

Table II.

Third-party questionnaire results; pre- and post-programme ratings

Question group	Z a	Asymp. significance (2-tailed)	Relative % increase
Understanding of healthcare systems and the structure of the NHS	−1.604b	p = 0.109	25
Knowledge and understanding of successful, safe and ethical decommissioning of services	−1.633b	p = 0.102	19
Knowledge and understanding of the crisis care concordat	−1.604b	p = 0.109	16
Communication skills	−1.342b	p = 0.180	14
Leadership skills	−2.121b	p = 0.034	19
Confidence	−1.633b	p = 0.102	15
Influencing skills	−2.121b	p = 0.034	21
Commissioning skills and knowledge	−1.841b	p = 0.066	20
Working effectively in partnership with other organisations	−2.121b	p = 0.034	20
Building and maintain working relationships	−1.890b	p = 0.059	16
Supporting others to improve performance	−2.121b	p = 0.034	17
Leading change through people	−2.121b	p = 0.034	21
Managing conflict	−1.857b	p = 0.063	18

Notes:

^a

Wilcoxon Signed Ranks Test;

^b

Based on negative ranks

Table III.

Validated questionnaire results: self-completion and third party

Question group	Self rating			Third party
	Z a	Asymp. significance (2-tailed)	Relative % increase	Z	Asymp. significance (2-tailed)	Relative % increase
Leadership skills	−2.536b	p = 0.010	36	−2.121b	p = 0.034	19
Influencing skills	−2.555b	p = 0.011	28	−2.121b	p = 0.034	21
Leading change through people	−2.236b	p = 0.025	15	−2.121b	p = 0.034	21
Supporting others to improve performance	−2.214b	p = 0.027	23	−2.121b	p = 0.034	17
Working effectively in partnership with other organisations	−2.636b	p = 0.008	22	−2.121b	p = 0.034	20

Notes:

^a

Wilcoxon Signed Ranks Test;

^b

Based on negative ranks