The good coach: implementation and sustainment factors that affect coaching as evidence-based intervention fidelity support

Lara M. Gunderson (Behavioral Health Research Center of the Southwest, Pacific Institute for Research and Evaluation, Albuquerque, New Mexico, USA)
Cathleen E. Willging (Behavioral Health Research Center of the Southwest, Pacific Institute for Research and Evaluation, Albuquerque, New Mexico, USA)
Elise M. Trott Jaramillo (Behavioral Health Research Center of the Southwest, Pacific Institute for Research and Evaluation, Albuquerque, New Mexico, USA)
Amy E. Green (Child and Adolescent Services Research Center, University of California, San Diego, California, USA)
Danielle L. Fettes (Child and Adolescent Services Research Center, University of California, San Diego, California, USA)
Debra B. Hecht (Section of Developmental and Behavioral Pediatrics, University of Oklahoma Health Sciences Center, Oklahoma City, Oklahoma, USA)
Gregory A. Aarons (Child and Adolescent Services Research Center, University of California, San Diego, California, USA)

Journal of Children's Services

ISSN: 1746-6660

Publication date: 19 March 2018

Abstract

Purpose

Evidence-based interventions (EBIs) for human services unfold within complicated social and organizational circumstances and are influenced by the attitudes and behaviors of diverse stakeholders situated within these environments. Coaching is commonly regarded as an effective strategy to support service providers in delivering EBIs and attaining high levels of fidelity over time. The purpose of this paper is to address a lacuna in research examining the factors influencing coaching, an important EBI support component.

Design/methodology/approach

The authors use the Exploration, Preparation, Implementation, and Sustainment framework to consider inner- and outer-context factors that affect coaching over time. This case study of coaching draws from a larger qualitative data set from three iterative investigations of implementation and sustainment of a home visitation program, SafeCare®. SafeCare is an EBI designed to reduce child neglect.

Findings

The authors elaborate on six major categories of findings derived from an iterative data coding and analysis process: perceptions of “good” and “bad” coaches by system sustainment status; coach as peer; in-house coaching capacity; intervention developer requirements vs other outer-context needs; outer-context support; and inner-context support.

Practical implications

Coaching is considered a key component for effective implementation of EBIs in public-sector systems, yet is under-studied. Understanding inner- and outer-context factors illuminates the ways they affect the capacity of coaches to support service delivery.

Originality/value

This paper demonstrates that coaching can accomplish more than provision of EBI fidelity support. Stakeholders characterized coaches as operating as boundary spanners who link inner and outer contexts to enable EBI implementation and sustainment.

Keywords

Citation

Gunderson, L., Willging, C., Trott Jaramillo, E., Green, A., Fettes, D., Hecht, D. and Aarons, G. (2018), "The good coach: implementation and sustainment factors that affect coaching as evidence-based intervention fidelity support", Journal of Children's Services, Vol. 13 No. 1, pp. 1-17. https://doi.org/10.1108/JCS-09-2017-0043

Download as .RIS

Publisher

:

Emerald Publishing Limited

Copyright © 2018, Emerald Publishing Limited


1. Introduction

Implementation of evidence-based interventions (EBIs) – health or human service interventions supported by rigorous scientific research – represents an increasingly accepted strategy for improving the wellbeing of families at risk for child maltreatment (Novins et al., 2013). However, it has long been known that the implementation of EBIs within day-to-day service delivery milieus is often inconsistent and ineffective, underscoring the challenges of transferring science to service (Horwitz et al., 2010; Novins et al., 2013). Researchers consider the implementation of innovative human service technologies, such as EBIs, to be more complicated than other types of technology, because they are delivered by individuals and organizations within complex, multi-layered social contexts (Aarons et al., 2011; Fixsen et al., 2009). In this paper, we consider how systematic “consultation,” or “coaching,” can assist providers and organizations in implementing and sustaining EBIs.

Our case study of coaching, defined here as ongoing support, or technical assistance, from a specialist to improve EBI implementation after training (Beidas et al., 2013; Duda and Barrett, 2013; Kucharczyk et al., 2012), draws from a larger qualitative data set documenting implementation and sustainment of a home visitation program, SafeCare®, an EBI to reduce child neglect (Chaffin, Hecht, Bard, Silovsky and Beasley, 2012; Gershater-Molko et al., 2003; Whitaker et al., 2012). Interviews and focus groups were conducted with government administrators and staff of community-based organizations (CBOs) contracted to deliver SafeCare in 11 United States human service (e.g. child welfare) systems. We examine how stakeholder perceptions and experiences of coaching changed during different phases of implementation, describe the significance of these changes, and discuss factors influencing coaching support.

Studies show that while training alone (i.e. “one-shot workshops”) is unlikely to result in practice change, let alone fidelity to an EBI (Odom, 2009), there are strategies to improve adherence to EBIs that range from didactic methods (Perepletchikova et al., 2007) to systematic review of care such as audit and feedback (Foy et al., 2005; Ivers et al., 2012) to observations performed in vivo or through recordings or one-way mirrors (Forgatch et al., 2005; Sheidow et al., 2008). Such strategies vary in effectiveness (Forgatch et al., 2005; Ivers et al., 2012). Studies of audit and feedback suggest limited evidence on reliability and how to best employ this approach, although intensive feedback appears to enhance effectiveness (Jamtvedt et al., 2006; Foy et al., 2005). Observation can better assess fidelity than self-report; continuous supervision, role plays, and periodic booster trainings may also reduce deviation from intervention protocols (Perepletchikova et al., 2007). Although potentially more costly, labor intensive, and intrusive to practitioner and client than other strategies, observation as part of fidelity monitoring can contribute to greater adherence to and positive outcomes for home visitation interventions (Forgatch et al., 2005; Weiss et al., 2006).

Coaching is an implementation strategy that may involve in vivo observation, intensive feedback, and other methods described above to support use of and fidelity to EBIs, including SafeCare (Chaffin, Bard, Bigfoot and Maher, 2012). Researchers point to the critical role that coaching can play in implementing and sustaining EBIs with fidelity by ensuring ongoing support after initial training (Duda and Barrett, 2013; Edmunds et al., 2013; Fixsen et al., 2009). Coaches supply implementation support, for example, with therapists delivering cognitive-behavioral therapy to youth (Beidas et al., 2013), by the National Professional Development Center on Autism Spectrum Disorders for improved EBI provision to students with autism (Kucharczyk et al., 2012), and for preschool teachers to promote school readiness for at risk students with the Children’s School Success Project (Odom, 2009; Odom et al., 2010). Recognized characteristics of effective coaches include abilities to negotiate diverse stakeholder needs in hierarchically structured organizations and service systems (Duda and Barrett, 2013), and to cultivate the attitudes and behaviors among frontline workers that are most likely to result in successful implementation (Kucharczyk et al., 2012). Adequately trained and supported coaches can build trust, develop productive relationships, and create accountability for EBI implementation outcomes (Duda and Barrett, 2013).

Although researchers suggest that coaching improves fidelity to EBIs, the specific strategies and interactions by which coaching contributes to desired outcomes is unclear (Beidas et al., 2013). EBIs are only as good as the support systems in place to ensure their integration into practice (Odom, 2009), and more research is needed to shed light on the nuances of implementing these systems (Fixsen et al., 2005, 2009). Our study offers a unique opportunity to examine coaching as a support system by exploring the situated perspectives of multiple stakeholders implementing the same EBI in varied service systems.

1.1 Conceptual framework

Several frameworks define implementation as a complicated process involving a series of stages and factors at multiple levels, i.e., system, organization, provider, and client (Aarons et al., 2011; Damschroder et al., 2009; Meyers et al., 2012). The Exploration, Preparation, Implementation, and sustainment (EPIS) model (Figure 1), developed for public service settings (e.g. child welfare, mental health care, substance abuse treatment), segments processes of instantiating an intervention into four phases: exploration (consideration of new innovations, evidence, and “fit” within service systems), preparation (planning for implementation), implementation (training and provision), and sustainment (maintaining with fidelity).

The EPIS model applies the analytic categories of “inner context” and “outer context” to illuminate the factors that can impact implementation and effectiveness of coaching across phases. Inner-context factors related to coaching include organizational-level variables, such as CBO leadership, organizational culture and climate, job demands, and workforce retention, as well as provider-level factors, such as disposition toward EBIs, work stress, and job autonomy (Novins et al., 2013). Outer-context factors are pertinent to the system level of the broader environment in which organizations operate (Aarons et al., 2011; Damschroder et al., 2009). A primary outer-context variable that may shape coaching is leadership in government-administered human service systems, which may determine the nature of contracts forged with CBOs to finance and facilitate EBI implementation, scopes of work embedded within contracts, allocation of training, coaching, or other support systems, and interorganizational relationships involving the sharing of resources (e.g. training and coaching) among CBOs.

Through this case study on the instantiation of a single EBI, we found that coaches play an important role in implementation because their activities span inner and outer contexts, influencing how system-level ideas and processes concerning EBI fidelity are translated and operationalized locally. Coaches are answerable to both frontline workers and CBO management in the inner context, and government administrators in the outer context. The present study elaborates on the range of inner- and outer-context factors that can impinge on coaching and its effectiveness over time, adding to our understanding of how coaching can support implementation and sustainment of EBIs more broadly within human service systems.

2. Methodology

2.1 Study context

This examination of coaching processes utilizes secondary analysis of qualitative data collected during three iterative mixed-method studies of SafeCare implementation and sustainment in one state-operated and ten county-operated human service systems in two states (Aarons, Fettes, Hurlburt, Palinkas, Gunderson, Willging and Chaffin, 2014; Aarons et al., 2009, 2012; Green et al., 2016; Willging et al., 2015). In most systems, state or county government agencies contracted with CBOs to deliver SafeCare, specifying both staffing and reporting requirements. SafeCare addresses vulnerabilities in families at risk or reported for child maltreatment by enhancing the parenting skills of primary caregivers. The contracts required CBOs to deliver three SafeCare modules on child health, home safety, and parent-child or parent-infant interaction. Staff called “home visitors,” typically case managers responsible for working directly with families, took part in a one-week training to administer the modules to caregivers through explanation, assigned homework, role play, and hands-on demonstrations.

Coaching is a key feature of SafeCare and other EBIs to ensure the intervention is delivered in the way research has found to be effective (McHugh and Barlow, 2010). Coaching for SafeCare is modeled on tenets of effective supervision and a developmental consultation approach (Stoltenberg, 2005; Stoltenberg and Delworth, 1987), and is required by the National SafeCare Training and Research Center (NSTRC), the entity that developed SafeCare. Coaches are the “support and fidelity monitoring system” for home visitors (Aarons et al., 2009). As per the NSTRC’s coaching manual, coaching is a “collaborative process” through which coaches help home visitors refine skills as they learn how to provide parenting feedback to families. Previous research on SafeCare found that in vivo coaching for fidelity monitoring can function as quality control, and enhance working alliances between home visitors and clients (Chaffin, Hecht, Bard, Silovsky and Beasley, 2012).

In all study systems, a subset of home visitors was trained to become coaches and tasked with accompanying other home visitors to see clients monthly, using a checklist to document adherence to each module’s core components. Thus, coaches were responsible for assessing fidelity and offering technical support. Each system designated a “lead coach” who communicated with the NSTRC, taught and certified coaches and home visitors in SafeCare, and provided implementation oversight. Coaches carried caseloads as home visitors and therefore also received coaching. In several systems, coaches convened as a group and with a lead coach for continued support. Some CBOs relied on personnel from other CBOs to coach home visitors. During the preparation and implementation phases, the NSTRC also made their trainers and coaches available to home visitors. These “external trainers and coaches” trained and certified home visitors locally, and helped CBO administrators select trainees to become internal coaches and trainers. Table I describes these roles.

2.2 Sample

For more than ten years, we conducted individual semi-structured interviews (n=166), small group interviews with an average of three participants (n=13), and focus groups with an average of six participants (n=80) with a range of stakeholders in all 11 service systems. System-level stakeholders involved in planning and managing services included government administrators (e.g. directors of child welfare agencies), CBO administrators (e.g. executive directors and program managers), and funders (e.g. directors of funding agencies). Stakeholders at the frontline were clinical supervisors of home visitation staff, SafeCare coaches, and home visitors. As shown in Table I, coaches were distinct from clinical supervisors, who oversaw the cases and overall wellbeing of families, but were usually not SafeCare trained and certified. However, four systems had individuals who performed the roles of both supervisor and coach. Most participants were women, particularly the frontline workers, as is common in helping professions (Willging et al., 2017). Participants generally reflected the cultural demographics of workers and the service populations in their respective systems. Ages ranged widely, and most held at least a Bachelor’s degree, with Master’s degrees more common among supervisors and higher-level administrators.

For this analysis, we classified systems by sustainment status as suggested by Stirman and colleagues (2012): “full” (n=7); “partial” (n=1); and “non-” (n=3). In full-sustainment systems, core implementation elements of SafeCare, i.e., ongoing delivery of modules, monthly coaching, and weekly team meetings, were maintained after initial implementation support had been withdrawn, and adequate capacity existed to maintain these elements. Partial sustainment described systems meeting only some elements after withdrawal of initial implementation guidance (e.g. service delivery continued but the system did not conduct model-required coaching). In non-sustaining systems, the EBI was no longer being provided by the system.

2.3 Data collection and analysis

Data were collected at three time points: Time 1 (T1; preparation/implementation phase; 2006-2008), Time 2 (T2; implementation phase; 2009-2011), and Time 3 (T3; sustainment phase; 2012-2014). Data were collected in at least one system each year across all three time periods. Individual systems began implementation at different times or had divergent trajectories of implementation; the bulk of data collection occurred in T3 when systems had been implementing the EBI for a minimum of two years. During T3, all coaching needs were being addressed by local stakeholders. Several stakeholders with long tenures in their positions were interviewed multiple times across time periods.

Interview and focus group guides in T1, T2, and T3 examined inner- and outer-context factors related to the preparation, implementation, and sustainment phases of SafeCare in each system, including the role of coaches and the influence of coaching on implementation and sustainment. Separate guides were developed for system and organizational stakeholders. Questions in T1 guides related to decision making to implement the EBI and first impressions of the EBI; T2 questions concerned impressions of training, coaching, and impacts of EBI implementation on systems, CBOs, and staff; and T3 questions centered on the role of leadership and collaboration in EBI implementation and sustainment, and experiences delivering and coaching the EBI. The University of California, San Diego Institutional Review Board approved the research design, sampling method, and consent procedures.

All interviews and focus groups were digitally recorded, professionally transcribed, and checked for accuracy by at least one author. We used NVivo 10 qualitative data analysis software (QSR International, 2012) for data management and to support iterative review and analysis. First, transcripts for each study in our database were coded by research assistants to condense the data into analyzable units. Codes were assigned segments of text ranging from a phrase to several paragraphs based a priori on the topic areas and questions making up the interview guides (Patton, 2015). For the secondary analysis, we re-reviewed and recoded the transcripts for content specific to coaches and coaching. As part of this secondary analysis, we created additional codes based on key sensitizing concepts from the implementation literature, including coaching or consultation, (clinical) supervision, and support. These concepts provided “a general sense of reference” for the secondary analysis and supplied descriptive data based in the actual words of research participants, which enabled us to examine their salience and meaning for different types of stakeholders over time (Patton, 2015, p. 545). We then used open and focused coding to locate new issues related to coaching in the transcripts (e.g. trainers, implementation support, scheduling coach visits) and to determine which issues emerged frequently or represented particular concerns of the stakeholders (Corbin and Strauss, 2008). By comparing and contrasting codes in both the primary and secondary analyses of the data, we grouped codes with similar content into broad categories of the findings linked to segments of text in our database (Corbin and Strauss, 2008; Glaser and Strauss, 1967). Within this database, we also incorporated detailed memos describing and linking codes to the larger categories, which, in turn, illuminated the role of coaches in implementing and sustaining the EBI and how different types of stakeholders perceived and experienced coaching. Finally, accuracy checks of the findings and assistance with interpreting data were provided by persons involved in organizing and implementing the coach role in study systems (Patton, 2015).

3. Results

The six major categories of findings are described below in a format that tells the story of SafeCare implementation across important EPIS phases, and in a way that underscores their connectedness: perceptions of “good” and “bad” coaches by system sustainment status; coach as peer; in-house coaching capacity; intervention developer requirements vs other outer-context needs; outer-context support; and inner-context support. Table II organizes results by sustainment status and EPIS phase to highlight how inner- and outer-context factors influenced coaching over time.

3.1 Perceptions of “good” and “bad” coaches by system sustainment status

Several characteristics of a “good” coach were identified during implementation by stakeholders in sustaining and partial-sustaining systems. Inner-context stakeholders deemed coaches effective when they validated and encouraged the work of home visitors and helped them advance their skills, often by modeling correct techniques: “They’re able to guide the staff in a way that supervisors aren’t,” stated one CBO administrator. Home visitors added that feedback was detailed and framed in a positive light. Stakeholders concurred that good coaches were “available,” “accessible,” and skilled at building “rapport” with clients and home visitors, paralleling techniques taught to home visitors in training, and as designed by the intervention developers. Describing her/his coach, a home visitor said, “[S/he] builds on [your] strengths so much and gets a twist on things you still have to work on.” Home visitors and supervisors, especially at the onset of implementation, also valued coaches for offering useful suggestions not covered in EBI materials, such as information about autism and domestic violence resources. Supervisors in sustaining systems clarified that good coaches possessed excellent organizational skills and kept track of complex schedules and great quantities of paperwork.

Coaches in sustaining and partial-sustaining systems referred to themselves as “problem-solvers,” “mentors,” and “helpers.” In contrast, government and CBO administrators commonly characterized the coaches as the “experts” charged with learning and teaching the EBI. This difference in understanding of the coaches’ role between inner- and outer-context stakeholders reflected the range of roles and responsibilities of participants. Home visitors and supervisors in the inner context depended on the coaches’ more quotidian technical support, while government administrators in the outer context emphasized the broader structure of implementation accountability that they perceived coaches as providing.

Stakeholders across systems shared similar understandings of characteristics associated with a “bad” coach. While home visitors were trained to think of coaches as peers, a bad coach was perceived as overly directive and a face of authority. They described bad coaches as making critical remarks, or correcting them in the presence of clients. One home visitor in a sustaining system shared, “I was taught that [coaches] are supposed to help us and to teach us and to support us, but I get criticized […] And in front of the clients, that’s even worse.” Another added that the occurrence of such scenarios causes clients to think, “This case manager doesn’t know how.” On the other hand, two home visitors agreed that while their coach was “energetic” about SafeCare, “[S/he] just displayed some really bad boundaries with my client.” A bad coach was also described as not available for or late to home visits, or as taking calls on their cell phone in the presence of clients. Bad coaches reportedly did not provide much if any feedback, leaving home visitors insecure about their fidelity ratings and their implementation ability and competence. A supervisor observed that when the coaches were “absent,” or not reachable by phone or in the office, the home visitors were unhappy with them.

Home visitors in non-sustaining systems experienced inconsistent and infrequent coaching, often due to turnover, a lack of in-house coaches to provide in-person support (see Section 3.3 below), and/or poor placement of individuals in coach and lead coach positions. The CBO administrators in one non-sustaining system recalled the home visitors feeling as though an external trainer/coach had demeaned them in a way that impacted implementation negatively. Sustaining and partial-sustaining systems reported similar descriptions but overcame this conflict through assiduous effort to complete the EBI training and with the help of other external trainers provided by the intervention developers. As shown in Table II, reports of bad coaching diminished over time as home visitors, coaches, and supervisors grew more comfortable with the EBI and as administrators learned what qualities make for a good coach.

3.2 Coach as peer

The EBI’s coaching manual defined coaching as the “support and monitoring system” for home visitors delivering SafeCare, but initial trainers during the preparation phase in the 11 systems emphasized coaches as complementary “peers,” engendering some confusion and discomfort among both coaches and home visitors. While coaches were trained to provide encouraging feedback, they were also charged with reviewing and addressing implementation fidelity, heightening anxieties among home visitors that coaches would judge their work performance negatively. One coach in a partial-sustaining system explained:

We started [implementing the EBI] at the same time. I don’t know more than they do. They don’t know more than I do. I don’t feel like that’s a good or fair position to put somebody in […]. We’re like equal coworkers so to be critiqued by your coworker is not a natural or comfortable situation.

Stakeholders in sustaining systems expressed similar perspectives. For example, one home visitor stated during implementation, “I’m sure a part of it is evaluating.” In a second system, two supervisors confessed to calling coaches “monitors” among themselves and said that they only use the term “coach” when in conversation with home visitors. A CBO administrator described coaching during implementation as “a role that’s a little bit confusing because the person is seen monitoring the work of their peers and yet they’re not in a supervisory position.” Stakeholders in non-sustaining systems did not reach an EPIS phase to comment extensively on coaches as peers.

This lack of clarity and consistency regarding the coaches’ supportive vs supervisory role was also commonly expressed by outer-context stakeholders. Some government administrators who lacked familiarity with the EBI treated, used, or talked about coaches as monitors. Government administrators who were not part of the exploration and preparation phases even sought to leverage the coach role to monitor the CBO home visitors and their billable hours. The coaches’ supervisors in one system reportedly called a meeting and explained to government administrators that coaches were not “supervisors or watchdogs.” One government administrator even wanted to become a coach specifically to monitor staff fidelity to the EBI. In these cases, outer-context stakeholders misunderstood or sought to recast the coaches’ supportive role in the inner context.

For their part, coaches were concerned with not looking like “bad guys” and consciously endeavored to frame comments to home visitors positively; those who had to coach supervisors especially expressed discomfort. Many felt more comfortable coaching home visitors from outside their training cohort. Confusion about the coach role was most ubiquitous during the preparation and implementation phases and more confined to new hires in the sustainment phase as stakeholders grew accustomed to coaching as part of service delivery and quality assurance.

3.3 In-house coaching capacity

Over the EPIS phases, it became clear in sustaining and partial-sustaining systems that home visitors benefitted from having their coaches housed within their own workplaces, rather than in another, possibly geographically distant, CBO. One home visitor remarked, “I do go to my coach when [s/he’s] there, but do I call to ask [her/him] something? No, I don’t. I call [the lead coach] or when I see the other coaches in the office, I ask them.” A lead coach in one system noted that, early in implementation, “turf wars” for referrals surfaced because coaches were only based in a single CBO, leaving some home visitors to believe that those with in-house coaches were favored within the system. According to this individual, tensions were reduced among CBO staff once coaches were finally housed in multiple CBOs.

Similarly, a government administrator in another system with geographically dispersed coaches wondered if the system had enough coaches while entering the sustainment phase and planned to ask the intervention developers for help. The participants in a focus group of coaches in this system agreed that it was harder to coach geographically dispersed home visitors, “We don’t really have that closeness with their management over there. We don’t have that much communication.” A government administrator in another sustaining system claimed that system stakeholders would not have agreed to do SafeCare without a local lead coach.

A minority of staff appreciated the opportunity to interact with coaches outside their CBOs, or to travel to new localities to perform coaching. For example, one home visitor said s/he liked having a coach from another CBO because the coach provided her/him with fresh ideas.

3.4 Intervention developer requirements vs other outer-context needs

Although government and CBO administrators were generally supportive of coaching, challenges in maintaining coaching capacity arose over time in many systems. In these cases, coaches were affected by the unintended consequences of policies, procedures, and funding arrangements originating in the outer context. For example, one CBO administrator instituted mandatory consultation times as some home visitors were not “eager” to schedule them. Government administrators in another system established a rule that home visitors must arrange coaching visits by the 15th of each month, to ameliorate rampant scheduling challenges. Coaches in this system said that, because they had to report those home visitors who did not schedule on time, resentment ensued among those who missed deadlines, undermining what was meant to be a supportive relationship. Home visitors also felt unfairly blamed for scheduling issues reportedly resulting from a reduction in coaching hours, which they believed were decreased by government administrators for financial reasons. Coaching became less consistent because of decreased hours, and coach participation in trainings for new home visitors was limited to save money, both of which fed into contradictory messages about the value accorded to coaching. The coaches believed that they no longer bonded as before with home visitors, nor were they privy to how training information was presented, hindering their ability to help home visitors.

In one system, the inability to maintain coaching in compliance with EBI requirements contributed to SafeCare’s failure to progress beyond implementation. Here, CBO administrators had trouble reconciling and integrating outer-context reporting requirements prescribed by the funder, local government, and intervention developer. The inability to efficiently report and bill for coaching reportedly led to fewer clients, and therefore lower levels of – and often unproductive – coaching. The failure to resolve the compliance issues around coaching contributed to CBO administrators deciding not to sustain the program.

3.5 Outer-context support

Administrators in the outer context affected the level of support for coaches in critical ways. For example, coaches wanted administrators to watch out for the danger of allocating large caseloads to home visitors that threatened fidelity. A coach during the sustainment phase asserted that, “If [home visitors] have any more than eight to ten cases I don’t feel that SafeCare is implemented in the way that it could be.” When asked about what was needed to sustain, one home visitor responded that they were dependent on “consultant and supervisor availability.” Finally, one lead coach asked to have her/his job requirements clarified and written down after experiencing greater and expanded responsibilities beyond what s/he perceived was appropriate.

Gaps in leadership in some systems caused by outer-context turnover during sustainment necessitated that coaches act as intermediaries between outer and inner contexts. Coaches said that they had to communicate between the CBO home visitors and the government administrators to facilitate continued sustainment. Home visitors and other stakeholders in some systems considered their coaches to be liaisons to administrators, often underscoring the pressures placed on them to span inner and outer contexts. One coach explained:

Sometimes I feel like we’re that middleman where we don’t have enough say sometimes to do certain things, and then sometimes they ask us for too much to do. So, it’s like we’re in that middle situation where we’re not their managers or their bosses but then sometimes they ask us questions that are for them.

Coaches benefited from having access to a coordinator dedicated to the SafeCare program, who simultaneously functioned as their lead coach; such individuals usually came from a supervisory administrative level. Coaches throughout implementation and sustainment phases characterized lead coaches as “key” to their ability to do their jobs. Government and CBO administrators also emphasized the importance of the lead coach during both phases, and some talked of having to replace an individual who was poorly performing in this position with a more “appropriate” person for effective implementation to progress. Lead coaches with multiple responsibilities (e.g. supervision, home visiting) commented in the sustainment phase that ideally, they would be exclusively coordinating the program in their vicinity. After multiple turnovers in the lead coach position, a CBO administrator in what became a non-sustaining system suggested that the organization should have specifically recruited for, rather than assigned existing staff to, the role.

3.6 Inner-context support

In addition to outer-context support, coaches were affected by the level of support they received within the inner context. A CBO administrator described having to communicate sternly to staff during implementation that their CBO was committed to EBIs:

[We had] a direct conversation, “We’ve gotta move forward.” You know, “We’ve gotta get on board. What do we need to do?” And so we changed how we were meeting with the consultant [coach]. We invited him to meetings. We changed our entire interactions […].

The CBO administration concentrated on changes that would explicitly support the coach, and thus EBI implementation.

Supervisors in the inner-context were particularly positioned to help or hinder coaches, according to lead coaches, coaches, and home visitors. Generally, supervisors were not trained in the EBI because administrators in both inner and outer contexts reasoned that supervisors could do their full-time administrative jobs without needing costly EBI-specific training. While some supervisors were trained with the intention of becoming coaches or attended portions of trainings when there was time or money for them to do so, most supervisors claimed that they did not have time to learn the model and that was a reason they valued the coaches. However, one coach underscored that it was the leadership of supervisors in the inner context that reinforced the coach role by helping home visitors understand and buy-into the implementation:

It makes a difference when you have a supervisor who understands, and who lets you know that, “This is what you have to do.” So, you have that support from the supervisor versus someone who just really doesn’t care. Then guess what? Your staff are not going to care either and they’re not going to do it and then that’s going to be my struggle.

Supervisors with a firm understanding of implementation processes greatly valued their coaches and how much the coaches helped them do their jobs better. Illustrating this point, one supervisor said that her/his peers were better able to oversee administrative procedures and could trust the coach to handle the EBI end of things.

However, supervisors also observed that coaches sometimes became tangled in a web of varied CBO policies and procedures when helping home visitors overcome implementation challenges. For example, there were instances when a coach suggested a solution that clashed with a CBO’s existing policies (e.g. no home visitations with families after dark). Supervisors in the sustainment phase were especially careful to keep issues of SafeCare implementation separate from their other work with home visitors. One observed, “Mixed messages can happen, that’s why I never speak about SafeCare. I immediately refer them back to their coach.” Coaches in a sustaining system elsewhere also recognized this overlay, with one explaining, “It’s because [our CBO has its] own set of rules. The other visitors may not follow the same guidelines so sometimes we get caught up in their management.” In these cases, coaches, particularly those not working in-house, struggled to adhere to outer-context requirements while managing inner-context realities. Over time, in sustaining systems, some home visitors became supervisors, drawing from their previous role, knowledge, and expertise to support the EBI. Lack of buy-in among supervisors reduced support for coaches in the inner context of systems unable to sustain.

4. Discussion

This qualitative research benefited from a series of interrelated studies that made it possible to longitudinally investigate a multitude of factors affecting coaching, a core support component of SafeCare and other EBIs (McHugh and Barlow, 2010). Our findings clarify factors helping and hindering coaching support for EBI implementation and sustainment in complex human service systems. The findings suggest that coaches perceived as successful were those who effectively navigated and negotiated shifting inner and outer contexts throughout the implementation phase and into the sustainment phase.

In sustaining systems, coaches enabled both implementation and fidelity, despite changing procedures, paperwork, and personnel. Coaches helped home visitors incorporate these changes and provided a source of constructive feedback and solutions to everyday challenges. However, our study also indicates that coaching can affect much more than intervention fidelity, as the efforts of coaches had implications beyond their defined role. Throughout the implementation and sustainment phases, for example, coaches were vital in communicating the intervention’s importance to home visitors, a notable aspect of organizational climate for EBI implementation (Aarons, Ehrhart, Farahnak and Sklar, 2014). In prior research, coaching also contributed to staff support for SafeCare and reduced staff turnover in child welfare systems (Aarons et al., 2012).

Our findings suggest a strong connection between home visitors’ positive evaluations of their coaches and the ability of these coaches to communicate constructively with them. In fact, the coaches perceived as “good” were practicing the principles of the EBI itself, such as inviting the home visitor to instruct the coach on his/her preferences (e.g. if s/he preferred the coach to model EBI techniques, participate, or simply observe), using specific vs general praise, and giving rationales for criticism. By operationalizing the same techniques that home visitors were to use with clients, coaches engaged in favorable consultation practices, a finding resonating with other implementation research (Beidas et al., 2013; Stoltenberg, 2005).

However, coaches in our study felt constrained in cultivating communicative and trusting relationships with home visitors by outer-context factors, such as when funding for their work hours was cut and they were unable to participate with home visitors in trainings. Home visitors also reported being less able to form trusting and supportive relationships with coaches who were not based within their daily workspace. Coaches, in turn, perceived that home visitors who received coaching remotely or did not have coaches nearby did not receive adequate assistance and demonstrated lower levels of EBI fidelity. This inner-context situation was an unintentional result of outer-context decisions about how to apportion limited funding, and perhaps the desire to control training and coaching integrity by locating coaches in one or a few organizations.

For their part, coaches depended on lead coaches and supervisors for support. Supervisors who understood the intervention and the intended role of coaches greatly influenced home visitor buy-in. In contrast, supervisors not on board with, or knowledgeable about, the EBI were not considered useful to home visitors, and in the worst cases, negatively influenced home visitor relationships with coaches and the EBI itself. This was problematic when home visitors received inconsistent messages about whether coaches were considered peers or monitors. In several systems, supervisors and government administrators, especially those unfamiliar with the EBI, sometimes treated, used, or talked about coaches as having only a fidelity monitoring function in a way that reinforced frontline anxieties about coaching. The negative effects of such misperceptions are consistent with scholarly understandings of the dimensions of effective implementation leadership (Aarons et al., 2015).

We recommend that the insecurities and initial turnover that affected home visitors can be diminished by government and CBO administrators proactively clarifying the coach role with the initial trainers during the preparation phase, and priming trainees to understand and embrace support from coaches (Aarons, Ehrhart and Farahnak, 2014). Furthermore, rather than rely on trial and error to hire coaches, greater knowledge of the role of coaching in the EBI during the preparation phase, and the characteristics of a good coach, will enable system and CBO administrators to recruit and select individuals for critical implementation positions who will support the EBI throughout the sustainment phase (Ehrhart et al., 2014; Schein, 2010) and increase fidelity. Additionally, recruitment, selection, and hiring of individuals with desirable characteristics to support implementation signify to employees the importance of the EBI (Ehrhart et al., 2014). This is in keeping with an implementation climate that communicates to employees what is expected, supported, and rewarded in organizations (Jacobs et al., 2014). Knowledgeable leaders can tailor the coach position to their implementation environments and balance it with other job assignments (e.g. supervisory functions) to prevent added responsibilities from presenting scheduling challenges like those that negatively affected the coaches in our study.

Our findings suggest that when leaders lacked knowledge of the appropriate role for coaches and expected them to act as monitors, coaches and their supervisors could resist such outer-context pressures by explaining to administrators that coaches were meant to serve in a supportive capacity. In situations like these, coaches and supervisors intersect with stakeholders differently positioned within system hierarchies (Duda and Barrett, 2013), thereby spanning inner and outer contexts. Coaches were often conceptualized by other stakeholders in this spanning capacity as communication liaisons. While having a liaison between inner and outer contexts was valuable for other system stakeholders, our study results suggest that the boundary-spanning position puts coaches at risk of being asked to fulfill duties above and beyond their intended role, such as reporting beyond fidelity assurance for the EBI. We suggest that turnover in leadership positions, funding issues, and changing relationships between CBOs and government contracting entities – common characteristics of human service systems – risk exacerbating this issue, and may compromise the implementation support coaches can provide to frontline workers. Consequently, we recommend that CBO managers be prepared to support and protect coaches as built-in quality assurance for EBI delivery by buffering them against additional demands. Furthermore, government administrators must ensure that contracted CBOs and coaches have the tools they need to best assess and augment coaching in their systems. This includes anticipating and proactively addressing funding changes, strategically applying resources for ongoing implementation, and directing sufficient provisions for enough coaches throughout the system (Willging et al., 2017; Aarons, Ehrhart and Farahnak, 2014).

This case study further demonstrates the importance of understanding the interaction between inner- and outer-context factors by illuminating the ways that they affect the capacity of coaches to support EBI implementation and sustainment. This builds on previous research that found evidence of outer-context decisions impacting inner-context processes (Willging, 2015, 2017). Our findings indicate that the effect of coaching on implementation depends largely on a coach’s ability to navigate, and help others navigate, dynamic service delivery environments; indeed, the context may be the primary mediator to consider for an EBI to progress to the sustainment phase (Dopson et al., 2008). In these environments, support components for EBIs, including coaching, work together and are limited in their ability to function independently to influence implementation (Fixsen et al., 2009; Wandersman et al., 2012). Leaders of CBOs and government agencies must be willing to collaborate to make organizational changes that strengthen the integration of coaching processes throughout EPIS phases (Aarons et al., 2011; Green et al., 2016). Ultimately, our study indicates that concepts of “good” and “bad” coaches relate to how well the coach role is integrated within CBOs, among home visitors, and within systems.

The confluence of multiple “embedding mechanisms” may create a strategic climate that promotes EBI use (Schein, 2010). Primary embedding mechanisms are strategies that leaders at multiple levels (e.g. system, CBO, and team or workgroup) can use to develop such climates. Examples include what leaders focus on regularly, how leaders react to crises, how they allocate resources and rewards, how they teach or role model, and how they recruit staff (Aarons, Ehrhart, Farahnak and Sklar, 2014). We contend that climates in which leaders at the system and organizational levels recognize the importance of coaching are likely to be conducive to EBI provision. Leaders demonstrate coaching support by conveying this importance to others, problem solving when coaching issues arise, capitalizing coaching appropriately, teaching others about, acknowledging, and rewarding coaches for their services, and being purposeful when selecting coaches.

For home visitors, stories about important events and people associated with the emergence and maintenance of coach roles may also inspire stakeholders to take coaching seriously, and perhaps to become coaches themselves. Stories that we were privy to throughout this investigation are themselves secondary embedding mechanisms that bolster support among system stakeholders for coaching as a crucial implementation quality control measure.

Other secondary embedding mechanisms concern how leaders design systems, organizations, and teams to support EBIs, including the development of reimbursement structures to compensate for coaching services, the sharing of coaching resources across CBOs, or how coaching-home visitation teams may be configured for fidelity monitoring and quality improvement purposes. This type of design can include formal statements highlighting the value of both the EBI and its support systems, i.e., coaching. Here, we recommend that government and CBO administrators clearly document descriptions of the coach position in service contracts and agency policies and procedures to maintain institutional memory that upholds the integrity of the coach role. This recommendation is consistent with implementation approaches that address the need for job redesign as part of comprehensive implementation strategies (Glisson and Schoenwald, 2005). Moreover, such actions will help mitigate the risk of coaches being blamed as “bad,” when, in fact, contextual factors may be culpable. Blaming the coach damages the support mechanism in place for the implementation. Documenting coach roles early may also ameliorate challenges related to turnover in government positions that led to new hires attempting to co-opt coaches as monitors. In our case study, communicating role expectations to all stakeholders may have diffused resentment among home visitors toward coaches when financial decisions led to issues with scheduling and inconsistent guidance.

Second, detailed agreements to fund ongoing training and coaching will help maintain the quality of coaching processes. Coaches who, for financial reasons, were reduced in training and billable coaching hours expressed concerns about having more distant relationships with home visitors and a limited ability to support them. A CBO in a non-sustaining system that was challenged to efficiently bill and report according to funder and government requirements may have sustained the implementation had they streamlined agreements first between the three entities.

Third, measuring and documenting processes and outcomes can create a climate for EBIs in systems and organizations (Aarons et al., 2011). Delineating reporting processes and consistent language agreed on by system stakeholders may ease communication challenges for coaches working across CBOs with different policies (Duda and Barrett, 2013), while simply describing coaches as peer support during training may generate confusion when coaches must rate the integrity of the implementation and provide guidance to home visitors. Adopting clearly articulated processes would also support coaches who become entangled in the different CBO policies and procedures when trying to solve implementation challenges.

5. Limitations

Although the systems where this research occurred were varied, the study took place in only two US states, thus limiting generalizability. Additionally, the study included participants from each non-sustaining system, but not all stakeholder groups were represented in later phases, as we could only access government and/or CBO administrators; coaches and home visitors no longer occupied the same positions related to the EBI. We also recognize that while our categorization of systems by sustainment status may allow us to suggest characteristics common to successful and less successful implementation environments, they may not fully capture the nuanced differences in implementation across systems. We are also limited in our capacity to compare across the categorizations, as the inclusion of only one partial-sustaining system limits understanding of the degree to which the elements of SafeCare were in play.

Whereas this case study emphasized the perceptions of coaching among different service delivery stakeholders, and we examined the SafeCare coach training manual, future research might include the clients who receive the intervention. Moreover, we contend that favorable components, characteristics, and procedures are dependent on local contextual factors. Nevertheless, this study benefits from 11 natural settings that enable us to examine in-depth the circumstances in which coaching occurs over time.

6. Conclusion

This paper elaborated outer- and inner-context factors affecting coaching in EBI implementation and sustainment. Our findings indicate that successful coaching involves trusting relationships and open communication, not only between coaches and home visitors, but also between coaches, outer-context stakeholders, and CBO administrators. Inner-context factors that facilitated EBI sustainment included CBO administrators enacting changes to better support coaches, use of in-house coaches, and buy-in among supervisors and home visitors. Outer-context factors facilitating sustainment included continued funding from government administrators, an understanding of the scope of work for coaches, and collaborating with CBO administrators to incorporate coaching and trainings into contracts.

Leaders in both outer and inner contexts can use embedding mechanisms to support coaching. In the present study, coaching can be considered an embedding mechanism signaling the importance of maintaining fidelity support as a critical part of EBI implementation. Where sites attained full sustainment, coaching and fidelity support were judged by the system and CBO leaders as critical to high-quality service delivery. Coaching, then, might be a valuable addition to other parenting interventions (e.g. Positive Parenting Program or Triple P) that may not emphasize this type of ongoing fidelity support at the level required by SafeCare (Chaffin, Hecht, Bard, Silovsky and Beasley, 2012; Seng et al., 2006; Funderburk et al., 2015). We recommend further institutionalizing the coach role through job redesign, clarification of coach requirements in job descriptions and performance evaluations, and recruitment for favorable coaching characteristics. Colocation of coaches in the offices of home visitors might also prove advantageous. Circulating stories about the positive role of coaching and coaches will likely reinforce coaching as a critical support system among stakeholders at multiple levels of a service system.

A clear contribution of this study is the identification of a common understanding of coaches as boundary spanners linking inner and outer contexts and facilitating communication to support implementation. However, while it may be valuable for coaches to span inner and outer contexts, doing so may also compromise the integrity of their specific role within EBI implementation and sustainment. Coaching supports EBI implementation and sustainment. Consequently, inner- and outer-context stakeholders must devote further attention to support coaching to improve fidelity and client outcomes in public-sector service systems.

Figures

EPIS implementation framework

Figure 1

EPIS implementation framework

Job roles

Personnel Function
CBO administrators Executive directors, area directors, program managers Oversee the general operations of CBOs, but are not directly involved in the day-to-day implementation of the EBI; provide administrative support to the agency and the EBI
CBO supervisors Team leader, frontline supervisor, clinical supervisor Responsible for general clinical and workplace management of home visitor teams. Some supervisors additionally serve multiple EBI-related roles as trainers and coaches
SafeCare trainers or lead coaches Persons trained and certified to train others in delivering the EBI. In the preparation phase, trainers were employed by the intervention developers rather than within the service systems Train home visitors in EBI practices during a 5-day workshop. Trainers or local lead coaches also coach home visitors after initial training. A minority of coaches are certified as trainers
SafeCare coaches Individuals trained and certified to coach others in delivering the EBI. Coaches are frontline supervisors or other direct service staff. In the preparation phase, intervention developer coaches were employed to start and scale-up the EBI until local coaches became certified Conduct monthly shadowing visits of each home visitor to monitor model fidelity and provide supportive coaching. The coaches serve as expert guides for home visitors for asking questions related to the EBI provision to clients
SafeCare home visitors Direct in-home service staff trained in the EBI Home visitors provide in-home EBI services to families

Coaching summary by sustainment status and EPIS phase

System Preparation (data from T1) Implementation (data from T2 and T3) Sustainment (data from T2 and T3)
Fully sustaining (n=7) Inner: CBO administrators, home visitors, and some supervisors learn about implementation and the coach role. CBOs experience turnover of veteran case workers in the transition to the EBI. Some home visitors express confusion about the coach’s advisory (vs supervisory) role. Some CBO staff are unhappy with not having an in-house coach Inner: CBO administrators make changes to better support coaches (e.g. mandatory consultation) and increase staff buy-in. Staff opinions of coaches improve but remain mixed. Scheduling challenges around coaching visits abound. Tensions exist between coaches and clinical supervisors regarding the coach role. Coaches depend on a lead coach for support Inner: coaches are generally integrated and valued throughout systems as part of quality assurance. Coaches maintain that it is awkward to coach peers and some supervisors remain confused about coaching role. Coaches continue to rely on lead coach
Outer: systems initiate EBI with external coaching and transition to their local team post-certification. Some external trainers are characterized by CBO staff as difficult, to the point of threatening EBI continuation Outer: communication issues with coaches and about coaching begin to resolve. Challenges of integrating coaching and the EBI into systems remain. Some systems expand coaching capacity as program grows Outer: government administrators appreciate coaches as providing built-in accountability, allowing them to focus on other work. Funding for coaching and ongoing trainings is a universal concern. Turnover in leadership requires some coaches to act as “boundary spanners”
Partial sustaining (n=1) Inner: CBO administrators, home visitors, and some supervisors learn about implementation and the coach role. There is discomfort at all levels with the idea of having a peer “coach” the other providers Inner: as with sustaining and non-sustaining systems, coaches claim discomfort coaching peers Inner: due to the lack of consistent coaching, home visitors are not implementing the EBI with full fidelity
Outer: external trainers are not well received Outer: the system loses one coach, leaving only the primary lead coach who refuses to travel to coach home visitors who are not nearby, leaving these more remote CBO staff without consistent coaching Outer: little support from government administrators in sharing resources after the loss of one coach
Non-sustaining (n=3) Inner: stakeholders learn about the implementation and the coach role. Staff express some resistance to the EBI Inner: coaching is inconsistent due to turnover and perception among home visitors that external coaching is unhelpful Inner: the EBI is no longer being implemented. CBO administrators note that adding the EBI to staff’s existing responsibilities posed challenges
Outer: some systems do not have local coaches and rely on external coaches. Multiple turnovers of lead coach position result in sporadic coaching in two systems Outer: systems insufficiently or inappropriately support coach or EBI due to financial and/or organizational issues. EBI implementation is negatively impacted by insufficient access to coaches Outer: government administrators state they should have written the contract with specific EBI needs and language in mind

References

Aarons, G.A., Ehrhart, M.G. and Farahnak, L.R. (2014), “The implementation leadership scale (ILS): development of a brief measure of unit level implementation leadership”, Implementation Science, Vol. 9, p. 45, available at: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-9-45

Aarons, G.A., Hurlburt, M. and Horwitz, S.M. (2011), “Advancing a conceptual model of evidence-based practice implementation in public service sectors”, Administration and Policy in Mental Health and Mental Health Services Research, Vol. 38 No. 1, pp. 4-23.

Aarons, G.A., Ehrhart, M.G., Farahnak, L.R. and Hurlburt, M.S. (2015), “Leadership and organizational change for implementation (LOCI): a randomized mixed method pilot study of a leadership and organization development intervention for evidence-based practice implementation”, Implementation Science, Vol. 10, p. 11, available at: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-014-0192-y

Aarons, G.A., Ehrhart, M.G., Farahnak, L.R. and Sklar, M. (2014), “Aligning leadership across systems and organizations to develop a strategic climate for evidence-based practice implementation”, Annual Review of Public Health, Vol. 35, pp. 255-74.

Aarons, G.A., Fettes, D.L., Sommerfeld, D.H. and Palinkas, L.A. (2012), “Mixed methods for implementation research: application to evidence-based practice implementation and staff turnover in community-based organizations providing child welfare services”, Child Maltreatment, Vol. 17 No. 1, pp. 67-79.

Aarons, G.A., Sommerfeld, D.H., Hecht, D.B., Silovsky, J.F. and Chaffin, M.J. (2009), “The impact of evidence-based practice implementation and fidelity monitoring on staff turnover: evidence for a protective effect”, Journal of Consulting and Clinical Psychology, Vol. 77 No. 2, pp. 270-80.

Aarons, G.A., Fettes, D.L., Hurlburt, M.S., Palinkas, L.A., Gunderson, L., Willging, C.E. and Chaffin, M.J. (2014), “Collaboration, negotiation, and coalescence for interagency-collaborative teams to scale-up evidence-based practice”, Journal of Clinical Child and Adolescent Psychology, Vol. 43 No. 6, pp. 915-28.

Beidas, R.S., Edmunds, J.M., Cannuscio, C.C., Gallagher, M., Downey, M.M. and Kendall, P.C. (2013), “Therapists perspectives on the effective elements of consultation following training”, Administration and Policy in Mental Health and Mental Health, Vol. 40 No. 6, pp. 507-17.

Chaffin, M., Bard, D., Bigfoot, D.S. and Maher, E.J. (2012), “Is a structured, manualized, evidence-based treatment protocol culturally competent and equivalently effective among American Indian parents in child welfare?”, Child Maltreatment, Vol. 17 No. 3, pp. 242-52.

Chaffin, M., Hecht, D., Bard, D., Silovsky, J.F. and Beasley, W.H. (2012), “A statewide trial of the SafeCare home-based services model with parents in Child Protective Services”, Pediatrics, Vol. 129 No. 3, pp. 509-15.

Corbin, J. and Strauss, A. (2008), Basics of Qualitative Research: Techniques and Procedures for Developing Grounded Theory, 3rd ed., Sage Publications, Thousand Oaks, CA.

Damschroder, L.J., Aron, D.C., Keith, R.E., Kirsh, S.R., Alexander, J.A. and Lowery, J.C. (2009), “Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science”, Implementation Science, Vol. 4, p. 50, available at: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-4-50

Dopson, S., Fitzgerald, L. and Ferlie, E. (2008), “Understanding change and innovation in healthcare settings: reconceptualizing the active role of context”, Journal of Change Management, Vol. 8 Nos 3-4, pp. 213-31.

Duda, M.A. and Barrett, S. (2013), Coaching for Competence and Impact Brief 1: Defining Coaching, Washington, DC, OSEP Center on Positive Behavioral Interventions & Supports (PBIS).

Edmunds, J.M., Kendall, P.C., Ringle, V.A., Read, K.L., Brodman, D.M., Pimentel, S.S. and Beidas, R.S. (2013), “An examination of behavioral rehearsal during consultation as a predictor of training outcomes”, Administration and Policy in Mental Health and Mental Health, Vol. 40 No. 6, pp. 456-66.

Ehrhart, M.G., Aarons, G.A. and Farahnak, L.R. (2014), “Assessing the organizational context for EBP implementation: the development and validity testing of the Implementation climate scale (ICS)”, Implementation Science, Vol. 9, p. 157, available at: https://implementationscience.biomedcentral.com/articles/10.1186/s13012-014-0157-1

Fixsen, D.L., Blase, K.A., Naoom, S.F. and Wallace, F. (2009), “Core implementation components”, Research on Social Work Practice, Vol. 19 No. 5, pp. 531-40.

Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman, R.M. and Wallace, F. (2005), Implementation Research: A Synthesis of the Literature, University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network, Tampa, FL.

Forgatch, M.S., Patterson, G.R. and Degarmo, D.S. (2005), “Evaluating fidelity: predictive validity for a measure of competent adherence to the Oregon model of parent management training”, Behavior Therapy, Vol. 36 No. 1, pp. 3-13.

Foy, R., Eccles, M.P., Jamtvedt, G., Young, J., Grimshaw, J.M. and Baker, R. (2005), “What do we know about how to do audit and feedback? Pitfalls in applying evidence from a systematic review”, BMC Health Services Research, Vol. 5, p. 50, available at: https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-5-50

Funderburk, B., Chaffin, M., Bard, E., Shanley, J., Bard, D. and Berliner, L. (2015), “Comparing client outcomes for two evidence-based treatment consultation strategies”, Journal of Clinical Child & Adolescent Psychology, Vol. 44 No. 5, pp. 730-41.

Gershater-Molko, R.M., Lutzker, J.R. and Wesch, D. (2003), “Project safecare: improving health, safety, and parenting skills in families reported for, and at-risk for child maltreatment”, Journal of Family Violence, Vol. 18 No. 6, pp. 377-86.

Glaser, B.G. and Strauss, A.L. (1967), The Discovery of Grounded Theory: Strategies for Qualitative Research, Aldine de Gruyter, New York, NY.

Glisson, C. and Schoenwald, S.K. (2005), “The ARC organizational and community intervention strategy for implementing evidence-based children’s mental health treatments”, Mental Health Services Research, Vol. 7 No. 4, pp. 243-59.

Green, A.E., Trott, E., Willging, C.E., Finn, N.K., Ehrhart, M.G. and Aarons, G.A. (2016), “The role of collaborations in sustaining an evidence-based intervention to reduce child neglect”, Child Abuse & Neglect, Vol. 53, pp. 4-16.

Horwitz, S.M., Chamberlain, P., Landsverk, J. and Mullican, C. (2010), “Improving the mental health of children in child welfare through the implementation of evidence-based parenting interventions”, Administration and Policy in Mental Health, Vol. 37 Nos 1-2, pp. 27-39.

Ivers, N., Jamtvedt, G., Flottorp, S., Young, J.M., Odgaard-Jensen, J., French, S.D., O’brien, M.A., Johansen, M., Grimshaw, J. and Oxman, A.D. (2012), “Audit and feedback: effects on professional practice and healthcare outcomes”, Cochrane Database Systematic Reviews, Vol. 13 No. 6.

Jacobs, S.R., Weiner, B.J. and Bunger, A.C. (2014), “Context matters: measuring implementation climate among individuals and groups”, Implementation Science, Vol. 9, p. 46, available at: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-9-46

Jamtvedt, G., Young, J.M., Kristoffersen, D.T., Thomson O’brien, M.A. and Oxman, A.D. (2006), “Audit and feedback: effects on professional practice and health care outcomes”, Cochrane Database Systematic Review, No. 3.

Kucharczyk, S., Shaw, E., Smith Myles, B., Sullivan, L., Szidon, K. and Tuchman-Ginsberg, L. (2012), Guidance & Coaching on Evidence-Based Practices for Learners with Autism Spectrum Disorders, The University of North Carolina, Frank Porter Graham Child Development Institute, National Professional Development Center on Autism Spectrum Disorders, Chapel Hill, NC.

Mchugh, R.K. and Barlow, D.H. (2010), “The dissemination and implementation of evidence-based psychological treatments: a review of current efforts”, American Psychologist, Vol. 65 No. 2, pp. 73-84.

Meyers, D.C., Durlak, J.A. and Wandersman, A. (2012), “The quality implementation framework: a synthesis of critical steps in the implementation process”, American Journal of Community Psychology, Vol. 50 Nos 3-4, pp. 462-80.

Novins, D.K., Green, A.E., Legha, R.K. and Aarons, G.A. (2013), “Dissemination and implementation of evidence-based practices for child and adolescent mental health: a systematic review”, Journal of the American Academy of Child & Adolescent Psychiatry, Vol. 52 No. 10, pp. 1009-25.

Odom, S.L. (2009), “The tie that binds: evidence-based practice, implementation science, and outcomes for children”, Topics in Early Childhood Special Education, Vol. 29 No. 1, pp. 53-61.

Odom, S.L., Fleming, K., Diamond, K., Lieber, J., Hanson, M., Butera, G., Horn, E., Palmer, S. and Marquis, J., Children’s School Success, P. (2010), “Examining different forms of implementation and in early childhood curriculum research”, Early Childhood Research Quarterly, Vol. 25 No. 3, pp. 314-28.

Patton, M.Q. (2015), Qualitative Research & Evaluation Methods: Integrating Theory and Methods, 4th ed., Sage Publications Inc., Thousand Oaks, CA.

Perepletchikova, F., Treat, T.A. and Kazdin, A.E. (2007), “Treatment integrity in psychotherapy research: analysis of the studies and examination of the associated factors”, Journal of Consulting and Clinical Psychology, Vol. 75 No. 6, pp. 829-41.

QSR International (2012), “NVivo qualitative data analysis software (Version 10)”, available at: www.qsrinternational.com/products_nvivo.aspx

Schein, E. (2010), Organizational Culture and Leadership, John Wiley and Sons, San Francisco, CA.

Seng, A.C., Prinz, R.J. and Sanders, M.R. (2006), “The role of training variables in effective dissemination of evidence-based parenting interventions”, International Journal of Mental Health Promotion, Vol. 8 No. 4, pp. 19-27.

Sheidow, A.J., Donohue, B.C., Hill, H.H., Henggeler, S.W. and Ford, J.D. (2008), “Development of an audio-tape review system for supporting adherence to an evidence-based treatment”, Professional Psychology, Research and Practice, Vol. 39 No. 5, pp. 553-60.

Stirman, S.W., Kimberly, J., Cook, N., Calloway, A., Castro, F. and Charns, M. (2012), “The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research”, Implementation Science, Vol. 7, p. 17, available at: https://implementationscience.biomedcentral.com/articles/10.1186/1748-5908-7-17

Stoltenberg, C.D. (2005), “Enhancing professional competence through developmental approaches to supervision”, American Psychologist, Vol. 60 No. 8, pp. 857-64.

Stoltenberg, C.D. and Delworth, U. (1987), Supervising Counselors and Therapists: A Developmental Approach, Jossey-Bass, San Francisco, CA.

Wandersman, A., Chien, V.H. and Katz, J. (2012), “Toward an evidence-based system for innovation support for implementing innovations with quality: tools, training, technical assistance, and quality assurance/quality improvement”, American Journal of Community Psychology, Vol. 50 Nos 3-4, pp. 445-59.

Weiss, H. and Klein, L., Hestia Advising (2006), Changing the Conversation about Home Visiting: Scaling up with Quality, Harvard Family Research Project, Cambridge, MA.

Whitaker, D.J., Ryan, K.A., Wild, R.C., Self-Brown, S., Lutzker, J.R., Shanley, J.R., Edwards, A.M., Mcfry, E.A., Moseley, C.N. and Hodges, A.E. (2012), “Initial implementation indicators from a statewide rollout of SafeCare within a child welfare system”, Child Maltreatment, Vol. 17 No. 1, pp. 96-101.

Willging, C.E., Green, A.E., Gunderson, L., Chaffin, M. and Aarons, G.A. (2015), “From a ‘perfect storm’ to ‘smooth sailing’: policymaker perspectives on implementation and sustainment of an evidence-based practice in two states”, Child Maltreatment, Vol. 20 No. 1, pp. 24-36.

Willging, C.E., Trott, E.M., Fettes, D.L., Gunderson, L., Green, A.E., Hurlburt, M.S. and Aarons, G.A. (2017), “Research-supported intervention and discretion among frontline workers implementing home visitation services”, Research on Social Work Practice, Vol. 27 No. 6, pp. 664-75.

Supplementary materials

JCS_13_1.pdf (2.8 MB)

Corresponding author

Dr Lara M. Gunderson is the corresponding author and can be contacted at: lgunderson@pire.org

About the authors

Dr Lara M. Gunderson is an Associate Research Scientist at the Pacific Institute for Research and Evaluation who studies implementation of evidence-based interventions in complex service systems. She received her PhD in the Department of Anthropology, University of New Mexico. Her dissertation research took place in Nicaragua where she also served in the Peace Corps as a community health educator.

Dr Cathleen E. Willging is a Senior Research Scientist and mental health services researcher at the Pacific Institute for Research and Evaluation. Her research centers on public mental health and substance use services in the USA, health care reform, interventions to reduce health and health care disparities, and implementation science.

Dr Elise M. Trott Jaramillo is an Associate Research Scientist at the Pacific Institute for Research and Evaluation who studies implementation of evidence-based interventions in complex service systems. She received a PhD degree from the University of New Mexico in the Department of Anthropology. Her dissertation research focused on water politics and community activism in Northern New Mexico and Albuquerque.

Dr Amy E. Green is a Clinical Psychologist and an Assistant Project Scientist at the University of California, San Diego and the Child and Adolescent Services Research Center. Dr Green has also had a long-standing interest in child and family mental health services research, with a focus on the impact of health policies. Her current research centers on the implementation, adaptation, and sustainment of evidence-based practices in public-sector settings for youth and families, with an emphasis on systemic and policy factors.

Dr Danielle L. Fettes is a Sociologist and an Assistant Project Scientist at the University of California, San Diego and the Child and Adolescent Services Research Center. Her academic career is embodied by a commitment to the wellbeing of children and youth, with a primary research agenda focusing on at-risk populations and key developmental outcomes, including mental health, educational attainment, and substance use.

Dr Debra B. Hecht is a Clinical Child Psychologist and an Associate Professor at the Center on Child Abuse and Neglect, University of Oklahoma Health Sciences Center. Dr Hecht’s clinical and research interests include the implementation of evidence-based practice into the field and the development and evaluation of appropriate assessment and treatment programs for children who have been abused and neglected in their families.

Dr Gregory A. Aarons is a Clinical and Organizational Psychologist and a Professor in the Department of Psychiatry, University of California, San Diego and the Director of the Child and Adolescent Services Research Center. Dr Aarons’ research focuses on implementation science with emphases on measurement, conceptual frameworks, management and organizational issues, quality of care, policy, and evidence-based practice in public-sector mental health, HIV, substance abuse treatment, and social services settings.