Measuring institutional transformation: a multifaceted assessment of a new faculty development program

Purpose – This study aims to understand the extent to which a faculty development program that includes a week-long course design experience followed by sustained support changes new faculty’s perceptions, beliefs and teaching practices. The authors employed the teacher professional knowledge and skill (TPK&S) framework and characteristics of effective educational development interventions to drive the program development, implementation and assessment. Design/methodology/approach –This study utilized amixedmethods approach. Data sources include pre-/ mid-/post-program responses to a validated survey, pre-/post-program course syllabi analyzed using a validated rubric and pre-/post-classroom observations collected using the Classroom Observation Protocol for Undergraduate STEM (COPUS) instrument. Findings – Findings indicate transformative effects for participants’ beliefs about their teaching and changes to their instructional practices. Significant and practical effects were observed across different portions of the program for increases in participants’ self-efficacy, endorsement of a conceptual change approach toward teaching and perceptions of institutional support. Participants produced more learning-focused syllabi and many moved toward more student-centered instructional approaches in their teaching practices. Research limitations/implications –Due to the voluntary nature of the new faculty development program, this study may have been limited by participant self-selection bias and differential sample sizes for the study’s individual measures. Future research should consider designs which maximize faculty participation in measurement across all data sources. Originality/value –This study addresses shortcomings in prior studieswhich utilized limited data sources to measure intervention impact and answers the call for more rigorous research to obtain a more complete picture of instructional development in higher education.


Introduction
An abundance of studies demonstrates the importance of implementing active learning in undergraduate courses (Freeman et al., 2014;Kuh et al., 2008). However, lecture still predominates in the majority of undergraduate classrooms (Stains et al., 2018). One reason for the limited adoption of active learning in higher education may be the use of educational Measuring institutional transformation development strategies that are ineffective in promoting changes in instructional practices (Henderson et al., 2011;Kezar et al., 2015;Pallas et al., 2017). These researchers suggest interventions should build upon best practices in professional development, align with research about effective teaching and address individual and institutional factors that may promote or inhibit change. However, there is a lack of rigorous research on these interventions to identify what works, what does not and why (Kezar, 2014;Stes et al., 2007) and limited use of theory to drive understandings in educational development interventions (Henderson et al., 2011). This present study aims to address the limitations identified in the literature by describing the design of an intervention for new faculty based on best practices in educational development and exploring its impact on faculty beliefs, perceptions and teaching practices.

Promoting change in higher education
Contrary to the required teacher preparation courses for K-12 teachers, postsecondary faculty typically receive no formal pedagogical training (Robert and Carlsen, 2017;Walker et al., 2008;Weidman et al., 2001). Many higher education institutions have teaching centers that offer programs to new and experienced faculty to support teaching development. Depending on the resources, center programing supports individuals and groups of faculty, and intervention formats encompass one-on-one teaching consultations, workshops, multiday institutes and year-long faculty learning communities (Sunal et al., 2001).
In a review of studies exploring the impact of different change strategies on science, technology, engineering and mathematics (STEM) faculty's instructional practice, Henderson et al. (2011) describe key features of effective educational development interventions and areas for improving the development and impact of these interventions. Further, the researchers conclude that studies often do not build "on prior empirical or theoretical work; and most published results claim success of the change strategy studied, but the evidence presented is often not strong" (p. 977). The present study addresses these limitations by developing an intervention based on the theoretical and empirical research and creating a multifaceted assessment approach that identifies changes in teaching beliefs and uses direct measures of changing instructional practices.
Theoretical and empirical research base for educational development interventions As part of their review, Henderson et al. (2011) identify three characteristics of effective educational development strategies. They must (a) address individual characteristics such as faculty's beliefs about teaching; (b) involve interventions that last at least one semester and (c) require understanding that colleges or universities are complex systems and designing a strategy that is appropriate for such a system. In order to shift instructional practices in higher education, interventions developed to promote change in practice should attend to both internal (i.e. beliefs) and external factors (i.e. context). We use the three characteristics of effective change described in Henderson's analytical review to frame the research underlying the development of the present study's intervention.
Critical factors in educational development. A recent report of the American Academy of Arts and Sciences claims that the attention of teaching centers is "overwhelmingly rooted in general pedagogical knowledge, and indifferent to specific disciplines and subjects and their distinctive concepts and ideas" (Pallas et al., 2017, p. 2). As we will show, this is not the case in more sustained interventions offered by many teaching centers. In this study, we use the concept of teacher professional knowledge and skill (TPK&S) to understand these factors, one of which is teaching beliefs that influence instructional practice. The TPK&S framework addresses the limitations of the original conceptualization of Shulman's (1986) general pedagogical knowledge (GPK) and pedagogical content knowledge (PCK). TPK&S consists of two main components: teacher professional knowledge bases (TPKB) and topic-specific professional knowledge (TSPK) (Gess-Newsome, 2015). JRIT TPKB includes, but is not limited to, general knowledge about assessment (e.g. knowledge of formative assessment and how to use it to drive instruction), content (e.g. knowledge of the disciplinary content), students (e.g. knowledge of student development and approaches to inclusive and equitable teaching) and curriculum (e.g. knowledge of course design processes) identified from the literature. In higher education development, ideas such as backward and integrated course design (Fink, 2013;Wiggins and McTighe, 2005), educative assessment (Huba and Freed, 2000;Wiggins, 1998), active learning (Bonwell and Eison, 1991), student motivation (Elliot et al., 2017;Schunk et al., 2007), inclusive teaching (Walton and Cohen, 2011;Burgstahler, 2015) and transparent assignments (Winkelmes et al., 2016) form the literature base for TPKB ( Figure 1).
As the name suggests, TPKB is the basis for but is also informed by TSPK. Similar to PCK, TSPK encompasses knowledge specific to a disciplinary topic and includes relevant teaching strategies (e.g. knowledge of effective practices that can be used in particular contexts), understanding of students (e.g. knowledge of student understanding around a course topic) and disciplinary practices (e.g. knowledge of disciplinary practices and big ideas). However, TSPK is fundamentally different from PCK in that "TSPK is canonical, generated by research or best practice, and can have a normative function in terms of what we want teachers to know about a topic-and context-specific instruction" (Gess-Newsome, 2015, p. 33 (Brookfield and Preskill, 2012;National Research Council, 2012;Lund and Stains, 2015;Nilson, 2016;Sternglass, 2017;Sunal et al., 2014), knowledge about student misconceptions and conceptual change strategies (Brown et al., 2018;McConnell et al., 2006), and knowledge of scientific and engineering practices (Carmel et al., 2019;National Research Council, 2012) and reflective and contemplative practice (Boud et al., 2013;Gunnlaugson et al., 2014) form the basis for TSPK.
The TPK&S model includes "amplifiers and filters," factors that influence how the teacher enacts TPKB and TSPK in practice. Higher education research suggests that faculty beliefs about teaching and professional identity are formed as students (Luft et al., 2004;Trede et al., 2012) and that those beliefs shape the decisions faculty make in their teaching (Brown et al., 2006;Robert and Carlsen, 2017). Additionally, external factors such as departmental and institutional support also play a role in faculty's pedagogical choices (Michael, 2007;Shadle et al., 2017;Sturtevant and Wheeler, 2019). As a result, faculty often hold beliefs that can be described as traditional: seeing their role in delivering content, favoring lecture over active learning and viewing students as passive consumers of knowledge. Since beliefs are important predictors for teaching practice, educational development initiatives focus on shifting instructors' orientation from traditional beliefs to increased alignment with evidencebased practice (Sunal et al., 2001).
The TPK&S framework described above was originally developed for K-12 teaching contexts; however, it has been applied to various higher education contexts to understand undergraduate STEM instruction (Auerbach and Andrews, 2018). To our knowledge, TPK&S has not been used as a framework for describing the design of educational development interventions.
Effective delivery formats of educational development interventions. When considering the importance of duration on effective change strategies (Henderson et al., 2011), the research shows that extended interventions such as week-long course design institutes (CDIs) are effective in shifting instructors' course design practices from a content focus to a learning focus (Palmer et al., 2016). When offered by university-wide centers for teaching and learning (CTLs) for an interdisciplinary group of faculty, these interventions largely focus on building TPKB. However, there are some CDI designs that build in opportunities for increasing TSPK through differentiated activities and grouping faculty according to discipline or shared pedagogical challenges.
The research also demonstrates that educational development interventions lasting at least one semester are more impactful than shorter delivery formats (Henderson et al., 2011;Sunal et al., 2001) and that sustained engagement through faculty learning communities (FLCs) provide access to teaching resources, collegial community and time and structure for exploring new pedagogical strategies (Daly, 2011;Lee, 2010;Meizlish et al., 2018). FLCs can have a disciplinary focus honing in on TSPK. However, even cross-disciplinary FLCs provide opportunities to cultivate both TPKB and TSPK through differentiated activities.
Finally, receiving formative feedback in the context of individual consultations has also been shown to enhance instructional practice (Finelli et al., 2011;Trower, 2012) and allows for learning of TPK&S. Interventions that use a combination of all three (CDIs, FLCs and individual consultations) have the potential to provide varied opportunities for improving both TPKB and TSPK. The present study aims to explore changes in faculty beliefs and practices following an educational development intervention that combines all three effective intervention strategies.
Aligning and designing system-appropriate intervention strategies. Alignment with the particularities of complex institutional contexts is important for the success of educational development interventions. In research-intensive institutions, faculty are rewarded primarily for research productivity. In addition, studies found that faculty struggle to maintain high productivity in both teaching and research (Fairweather, 2008). This is in tension with an JRIT increased scrutiny of educational outcomes (Boyer Commission, 1998) and a call to close the education gap for marginalized populations (Ladson-Billings, 2006). In this constellation, research-intensive institutions need to reconsider reward structures and support effective, equitable and efficient teaching practices, particularly for new faculty who are expected to be highly productive in order to obtain tenure. Efforts directed to train new faculty, particularly in times of high faculty renewal, present institutions with the opportunity to reset the culture of teaching and learning. Furthermore, research shows that focusing on new faculty can be particularly impactful (Beach et al., 2016) as they may be more open to adopting effective teaching practices than more established colleagues (Ebert-May et al., 2011). In the present study, we address this call through the development and assessment of an educational development intervention for new faculty.

Assessment of educational development interventions
Although studies on educational development interventions have demonstrated their impact (Cilliers and Herman, 2010;Meizlish et al., 2018), assessment efforts that systematically move beyond participant satisfaction and perceptions are scarce. For example, published reports of faculty development initiatives often provide descriptive overviews (Sorcinelli et al., 2006), make inferences from program participation and levels of participant satisfaction (Amundsen and Wilson, 2012) or rely on limited sources of evidence to assess program effectiveness (Cilliers and Herman, 2010).
These shortfalls, however, are not surprising. Charged with developing and facilitating such programs, CTLs have historically had limited resources to conduct research on their interventions (Chism et al., 2012;Hines, 2009Hines, , 2011. In addition, the challenges of tracing the path from intervention to changes in teaching practices to improvements in student learning are "legion" as Mary Taylor Huber reminds us (Condon et al., 2016). In the present study, we aim to expand the literature by reporting on the design of an educational development initiative and findings from a multipronged assessment study to measure its impact.

Purpose
This article addresses limitations found in the previous research reviewed above. First, it responds to the call that theoretical and empirical research be used to develop and implement an educational development intervention by describing a research-based intervention design (Henderson et al., 2011). Second, the study on the intervention's impact addresses gaps in program assessment by reporting on the design of and findings generated by a multifaceted assessment approach.

Design of a research-based intervention (The New Faculty Program)
The New Faculty Program was created in 2015 to prepare large numbers of newly hired faculty during their first three years of teaching. It included an intensive, week-long CDI (35 contact hours) and a semester or year-long FLC (18 contact hours). This design was aligned with research on professional development, effective teaching and change strategies. It seeks to change faculty's beliefs about teaching through extended engagement over the course of at least one semester. It also utilized a learning community to support change and employed strategies that are largely accepted in our institution. Further, it aligned with the TPK&S framework, seen in Figure 2 and described below.
Course design institute. The intensive week-long course design boot camp was based on higher education research and aligned with the TPK&S framework to support faculty in designing evidence-based, learning-focused courses (see supplemental appendix for CDI overview). While the institute largely focused on TPKB that cuts across disciplines, it was also structured to continuously engage faculty in exploring PCK specific to their fields Measuring institutional transformation (i.e. TSPK). For example, participants were grouped according to disciplines (e.g. science, humanities) or specific pedagogical challenges or interests (e.g. contemplative pedagogy, community engagement, technology-enhanced learning). Further, self-guided learning activities throughout the week offered choices and discipline-specific examples. Thus, our CDI provided instructors opportunities to improve their TPKB and TSPK.
Faculty learning community. After CDI, program participants engaged in a semester-or year-long FLC consisting of a half-day retreat, eight 90-min meetings and an hour-long individual teaching consultation (see supplemental appendix for details on the FLC). The FLC topics and structure supported participants' translation of TPKB and TSPK into their course instruction. This translation was done both through program assignments and FLC meeting interactions. For example, before an FLC meeting on assessment, participants selected a formative learning assessment technique (Angelo and Cross, 2005;Barkley and Major, 2015) and implemented it in their course. During the FLC meeting, the faculty discussed with their peers how to refine the technique for the future. As another example, participants read a chapter on metacognition prior to an FLC meeting (Nilson, 2016) and selected a metacognitive strategy appropriate to one of their class objectives. They then created a plan for implementing the strategy in their course to improve students' metacognitive skills. During the FLC meeting, participants were grouped according to class size and provided each other feedback on the plans. In the following meeting two weeks later, participants reported on the successes and failures of implementation and received additional feedback from the group.

Program assessment and research questions
The present study aims to understand the extent to which the previously described educational development intervention changes new faculty's perceptions, beliefs (i.e. amplifiers and filters of TPKB and TSPK) and practices. Through a multifaceted

Higher education TPKB components:
Motivation & metacognition Formative/summative assessments Student prior knowledge Activities that promoted TPSK learning: Planning, implementation, discussion, and reflection on teaching strategies implemented in practice Figure 2. Alignment of the program with TPK&S framework. TPKB components listed stem from higher education research and practice. TSPK activities stem from research and practice in professional development and educational development JRIT assessment approach detailed below, the study seeks to address gaps in program assessment literature. The research questions for the study are as follows: ( 1) To what extent does participation in the New Faculty Program improve participants' self-perceived confidence with and attitudes toward learning-focused, evidence-based teaching practices?
(2) How does participation in the New Faculty Program impact instructors' perception of institutional support for teaching?
( 3) To what extent does participation in the New Faculty Program increase instructors' use of learning-focused course design principles and evidence-based teaching practices?

Methods
This mixed methods study was conducted at a research-intensive public institution in the Mid-Atlantic region of the USA (see appendix A for student and faculty demographic composition). Data sources include three surveys, course syllabi and observations.

Participants
Study participants (n 5 105) came from the first three cohort years (2015-2017) of the program. All participants were faculty who were employed by the university for no more than three years. Participation in the New Faculty Program was voluntary but strongly encouraged by the university's administration. Faculty were also encouraged to participate by word-of-mouth recommendations from former participants and the content of the New Faculty Program's description and curriculum (see supplemental appendix). A chi-squared goodness-of-fit test was conducted to determine the program's representativeness of the new faculty population during the years of their eligibility for enrollment in the New Faculty Program (2012)(2013)(2014)(2015)(2016)(2017). Analyses determined that program faculty were statistically equivalent to new faculty in their race/ethnicity and gender demographics (p > 0.05) (Appendix A) but differed statistically in their faculty rank, tenure status and discipline area. New faculty in the New Faculty Program were predominantly from the STEM disciplines and had the faculty rank of assistant professor (Table 1), while new university faculty were predominantly from the humanities disciplines and had a more balanced representation among assistant professor and instructor/lecturer ranks [1].

Measuring institutional transformation
Data sources and collection As aligned with the research questions, our data sources aimed to capture evidence of change in the amplifiers and filters that, according to the TPK&S framework, may impact the ways in which instructors teach. Survey data were collected from participants via Qualtrics at three different times: just before joining their program (pre-program), after completing their course design experience (post-CDI) and at the completion of their FLC program (post-program). Syllabi and classroom observations were collected for participants pre-and post-program (see Table 2 for an overview of the data collection process). Participation in the program's developmental and research components were both strictly voluntary for new faculty at the university. Faculty could elect to participate or not in any or all stages of data collection throughout the study. Consequently, due to the voluntary nature of study participation, we gathered a limited number of complete pre-/post data in some areas of assessment. Sample sizes ranged from a high of 53 (50.48% of participants) for one measure (syllabus analysis) to a low of 23 (21.90% of participants) for another measure (classroom observations). We address these differential participation rates in our findings and discussion. Incomplete responses from participants for specific assessments were excluded from the dataset.
Surveys. Surveys consisted of three instruments to assess participants' teaching selfefficacy (teaching appraisal inventory [TAI], Balam, 2006), their attitudes toward teaching (revised approaches to teaching inventory [ATI], Trigwell et al., 2005) and their sense of belonging to the institution (adapted from the classroom community scale [CCS], Rovai, 2002) (Appendix B).
Part A of the TAI, consisting of 43 Likert scale items, was used to measure participant teaching self-efficacy. The TAI consists of seven dimensions: assessment, class facilitation, effective assignments, goals and objectives, learning activities, learning environment and overall teaching.
The revised ATI, consisting of two independent 11-item scales, was used to assess the way participants go about teaching in a specific context, subject or course. The conceptual change/student-focused (CCSF) scale consists of questions around the idea that quality learning occurs when students change their conceptions of phenomena. The information transmission/teacher-focused (ITTF) scale consists of questions around the idea that effective learning occurs when knowledge is transmitted from teachers to students.
A revised version [2] of the CCS was used to measure participants' sense of community or belonging at the university. This 20-item instrument consists of two subscales with ten items each that assess faculty's connectedness and learning beliefs. The connectedness subscale measures feelings of "connectedness, cohesion, spirit, trust, and interdependence," while the learning subscale measures feelings of shared "values and beliefs concerning the extent to which their educational goals and expectations are being satisfied" (Rovai, 2002, pp. 206-207). Qualitative data were collected in an end-of-program questionnaire that assessed participants' beliefs about continued institutional support of their pedagogical development.

JRIT
They were asked to respond to the open-ended question: "Do you have any additional comments, concerns, or suggestions regarding support for your teaching after the New Faculty Program?" Additional qualitative data were gathered from commentary by the university's vice president and provost about the importance of the New Faculty Program and the institution's culture of teaching and learning.
All dimensions for the TAI (seven dimensions), ATI (two dimensions) and CCS (two dimensions) have Cronbach's alpha (α) values > 0.75, demonstrating acceptable to excellent levels of internal consistency (DeVellis, 2016). Based on prior studies that employ these instruments, individual questions were either averaged (TAI) or summed (ATI and CCS) for each dimension. We do not report on changes by individual items.
Course syllabi. For each participants' pre-and post-program syllabus, we used a rubric (Palmer et al., 2014) to assess the course's learning-focused orientation represented in the syllabus. The rubric includes 13 components within four main dimensions: Learning goals and objectives, assessment activities, schedule and the overall learning environment of promise, tone and inclusivity. A total of two raters independently scored each syllabus on the 13 components, compared their scores and discussed differences until consensus was reached. Each syllabus received an average score from the two raters on a scale from 1 (content focused) to 46 (learning focused). [In contrast to traditional contractual syllabi that focus on content coverage, grading procedures and policies, learning-focused syllabi clearly communicate what students will gain from the course, what they will do to achieve the course objectives, how they will be evaluated and how to best study and seek support].
Classroom observations. The Classroom Observation Protocol for Undergraduate STEM (COPUS) (Smith et al., 2013) was used to determine the presence or absence of 12 instructor and 13 student behaviors that occurred during 2-min time intervals over the course of the allotted class time. Observers, trained on COPUS [3], observed participants' instruction a minimum of once pre-program and once post-program.

Data analysis
Post hoc power analyses were performed using G*Power to determine if sample sizes were sufficient to detect a large effect size (r 5 0.5) with an alpha value set at 0.05. When there was insufficient power (<0.80), the power analysis was reported and descriptive statistics were used. Data were also tested to determine if assumptions of parametric testing were met. When assumptions were not met, appropriate nonparametric analyses were performed.
Surveys. We sought to identify changes in participants' seven dimensions of selfefficacy, two dimensions of teaching approaches and two dimensions related to sense of belonging across time. A one-way repeated measures ANOVA was conducted to identify these changes between the three time points (pre-program, post-CDI, post-program). We calculated effect sizes (η 2 p ) for each significant change observed, and post hoc comparisons using the Bonferroni correction identified specific differences between the three time points.
Course syllabi. Normalized gains were calculated for each participant's pre-/post-program syllabi scores using Hake's (1998) formula: <g> 5 100*(postpre)/(46pre), where 46 is the maximum score possible. Each syllabus also received a classification as content focused (0-16.5), transitional (17-30.5) or learning focused (31-46). A Wilcoxon signed-rank test was performed to test the significance of differences for total and criterion-level pre-and postprogram scores because data were not normally distributed. Effect sizes were calculated (r 5 z/√N), as suggested by Fritz et al. (2012).
Classroom observations. From the COPUS data gathered, % total time was calculated for each of the 25 behaviors. For example, if the observer coded for the presence of lecture for 20 of the 50 min of class, the % total time spent lecturing would be 40%. These percentages were then analyzed using the COPUS Analyzer (http://www.copusprofiles.org/) to generate a Measuring institutional transformation distinct instructional profile characterizing instructional practice as didactic (>80% lecture), interactive lecture (lecture with some group work) or student centered (group or individual activities used throughout class) (Stains et al., 2018).

Results
Below we report the data organized by research question. Additional statistical tables on post hoc Bonferoni pairwise comparisons can be found in Appendix C.

Changes in teaching self-efficacy and attitudes
Results indicate statistically significant (p < 0.05) mean score differences for all seven TAI dimensions of teaching self-efficacy (Figure 3), with effect sizes (η 2 p ) ranging from 0.232 to 0.294, demonstrating large practical effects (Cohen, 1988;Norouzian and Plonsky, 2017) for participants in all areas. Post hoc Bonferoni pairwise comparisons revealed significant gains (p < 0.05) across all seven self-efficacy dimensions between pre-program and post-CDI and between pre-program and post-program; however, no significant changes occurred in any of the seven dimensions post-CDI to post-program. Thus, participants' self-efficacy increased following CDI and was maintained across the FLC.
Results also indicate significant change in participants' mean scores on the conceptual change dimension (CCSF) of the ATI, F (2,72) 5 7.55, p 5 0.001, η 2 p 5 0.173 (Table 3). Post hoc Bonferoni pairwise comparisons demonstrate that participants made significant gains preprogram to post-CDI and pre-to post-program. However, there was no significant change 33.14 6.89 33.46 6.41 34.28 6.34 Note(s): * significant from pre-program, p < 0.05; ** significant from pre-program, p < 0.01. Each scale includes 11 individual Likert questions ranging from 1 5 rarely to 5 5 almost always (n 5 37)  Table 3. Changes in participants' approaches toward teaching JRIT between post-CDI and post-program. Similar to participants' self-efficacy scores, participants' attitudes toward student-focused instruction significantly improved following CDI and were maintained across the FLC.

Changes in course design and teaching practices
Pre-program syllabus total scores ranged from 0 to 43 (46 maximum points) with a mean score of 12.75 (SD 5 11.46). The majority of pre-program syllabi fell into the content-focused category (Table 4). Post-program syllabi total scores ranged from 0.5 to 46 points, with a mean score of 30.26 (SD 5 10.16). Results indicated that overall syllabi scores were significantly higher post-program compared to pre-program (z 5 5.96, p < 0.001, r 5 0.58). These significant improvements were observed for all four syllabus subcomponents (i.e. learning goals, assessment activities, schedule and learning environment). When exploring participants' pre-post-program normalized gains, mean gains were 49.5% (SD 5 33.4%). In other words, participants, on average, gained half of the possible points post-program based on their pre-program scores. Additionally, 13 participants achieved low gains (<g> ≤ 30%), 25 participants achieved moderate gains (30% < <g> < 70%) and 15 participants achieved high gains (70% ≤ <g> ≤ 100%).
Pre-program, nearly half (n 5 11, 47.8%) of participants' observed instructional practices were characterized as didactic (>80% lecture). Nearly another half (n 5 10, 43.5%) of participants' pre-program instructional practices were characterized as interactive lecture. The remaining three participants (8.7%) were observed using student-centered instructional practices pre-program, characterized by group work integrated throughout the class. Postprogram, ten (43.5%) participants' instructional practices were didactic, seven (30.4%) participants utilized interactive lecture and six (26.1%) participants were observed implementing student-centered instruction. When exploring individual changes in participants' instructional practices pre-and post-program, the majority (n 5 12, 52.2%) of participants' instructional practices did not change (Figure 4). However, of those participants that did shift, more shifted toward student-centered instruction (n 5 8, 34.8%) than to didactic instruction (n 5 3, 8.7%).
The percentage of total class time instructors spent on specific activities was compared pre-and post-program (Table 5). While descriptive in nature, participants appeared to spend less time lecturing and more time engaging students in group work and clicker questions.

Change in perception of institutional support
There was a significant increase in participants' sense of being connected to the institutional community F (2,48) 5 6.60, p 5 0.003, η 2 p 5 0.216, with an increase in mean scores over time ( Measuring institutional transformation supported by the institution was also observed F (2,48) 5 5.93, p 5 0.005, η 2 p 5 0.198, with mean scores also increasing over time. Post hoc pairwise comparisons demonstrated that participants felt significantly more connected to the institution pre-program to post-CDI and pre-to post-program; however, no significant changes occurred post-CDI to post-program. Participants' sense of the institutional support available for learning about teaching significantly improved pre-to post-program and post-CDI to post-program.
Participants generally expressed a belief that their continued pedagogical development was valued and would be supported by the university. One participant stated, In the past (primarily at another institution) I felt very isolated as I tried to grow as a teacher. At some point I had a goal to try something new in every class I taught, but I got overwhelmed, sometimes lazy, and unsupported by my colleagues. While I already felt much more supported as a teacher in my department here at the university, being in the New Faculty Program has really cultivated my enthusiasm for working toward being a great teacher. .67 Note(s): * significant from pre-program, p < 0.05; ** significant from pre-program, p < 0.01; þ significantly different from post-program, p < 0.05 (n 5 25). Each scale includes ten individual Likert questions ranging from 0 5 strongly disagree to 4 5 strongly agree Table 5. COPUS profile criteria changes  Table 6. Overview of participants' perceptions of institutional support JRIT Another participant commented on her new willingness to seek support from within her department: "I have begun to talk with colleagues more about their approach to teaching and discuss some of the challenges I have with my own class." Support from the university's administration was clear and encouraging in a video endorsement of the New Faculty Program's value. In the video, the vice president and provost stated, As a new faculty myself and getting ready to teach my first course in the fall, I'm excited to participate in the Ignite Program and get that added insight to be the best teacher I can be, but also to be a part of what makes the culture of teaching and learning so special here at UVA. It is something different from the rest of our peers, and you should join and be a part of it (CTE UVa, 2016).

Discussion
In summary, participants' teaching self-efficacy, attitudes about teaching with a conceptual change approach (CCSF) and connectedness to the university all significantly improved following CDI and were maintained across the New Faculty Program's FLC. Participants' perceptions of institutional support for learning about teaching significantly improved over the course of the FLC. Participants designed more learning-focused courses as evidenced by their more learning-focused syllabi. Finally, due to the small sample size of faculty participating in classroom observations, we were not able to draw conclusion about changes in instructional practices.

Instructor self-efficacy and attitudes
Participants' self-efficacy was measured across seven important dimensions for implementing learning-focused practices. Significant increases in faculty's self-efficacy occurred following their CDI experience, with gains maintained over the course of the FLC as they integrated these new learning-focused practices into their teaching. Early maintenance of teaching self-efficacy gains is remarkable and runs counter to the well-documented pattern where teachers' self-efficacy increases during skill acquisition and then dips during implementation when new skills are incorporated into their professional practices (Favre and Knight, 2016;Tschannen-Moran and McMaster, 2009;Woolfolk Hoy and Burke Spero, 2005). Rogan (2007) suggests that FLCs provide scaffolding and support during the crucial time when instructors integrate innovations into practice. Given that reflective practice is central to faculty teaching (Kane et al., 2004), we hypothesize that the opportunity to reflect upon and discuss their experiences during the implementation phase of our FLC may have offered faculty enough support to insulate them from the predictable dip in self-efficacy during their initial attempts to integrate new teaching practices. Furthermore, the New Faculty Program may have provided faculty with opportunities to translate TPKB and TSPK into practice through targeted assignments and reflection. Future research exploring the reflective practice of participating faculty could provide insight into how TPKB and TSPK interact and reinforce each other. More generally, research comparing the reflective practices of participating and nonparticipating faculty is also warranted.
Participant responses on the ATI indicate an increase in their endorsement of a conceptual change approach to teaching after their CDI experience. These gains were also maintained through their continued participation in the FLC. This outcome contradicts a recent claim that faculty development programs have minimal impact on their endorsement of a conceptual change approach and may actually promote a stronger endorsement of an information transmission approach ( € Odalen et al., 2019). Rather, our findings are consistent with earlier research demonstrating the efficacy of faculty development for increasing adoption of student-centered attitudes toward instruction (Gibbs and Coffey, 2004;Hanbury et al., 2008) and that other factors beyond pedagogical content knowledge and skill influence practice (Gess-Newsome et al., 2019).

Measuring institutional transformation
Participants in this study did not change their endorsement of the informational transmission approach. Mean scores remained stable and significantly lower than the mean scores for the conceptual change approach. The concepts connected to this approach were not supported by any of the learning-focused curriculum of the New Faculty Program. The observed stability of this approach along with the noted increase for endorsing the conceptual change approach in this study supports the independence of these two dimensions in the ATI.

Instructional practices
In our syllabus analysis, participants' overall normalized gain (<<g>>) of 49.5% following the New Faculty Program translates to an average gain of 22.8 points for faculty scoring the lowest points possible for a content-focused syllabus (0.0) and an average gain of 14.4 points for those scoring the lowest points possible for a transitional syllabus (17.0). In other words, the average participant in this study was expected to move up one category along the contentfocused to learning-focused syllabi continuum. This outcome compares to an earlier larger scale study of the impact of CDI on faculty of all ranks and levels of experience (Palmer et al., 2014), where <<g>> was reported at 60.4% (SD 5 22.4%). The normalized gains for program participants were slightly lower. We speculate that new faculty may have had more exposure to learning-focused teaching practices and may achieve less gains than more seasoned faculty who were less likely to be inducted into their careers with these methods.
Classroom observations conducted with a limited sample of program participants (n 5 23) revealed that participants appeared to spend less time lecturing and more time engaging students in group work and clicker questions. While descriptive in nature, nearly half of the faculty observed moved to more student-centered instructional approaches following the New Faculty Program, while a slight majority of faculty's practice did not shift, and three faculty members shifted from an interactive lecture to a didactic format. This variation is not surprising as Stains et al. (2018) suggest at least four classroom observations per participant to obtain an accurate assessment of their COPUS profile and repertoire of teaching strategies. Thus, more observations would need to be conducted for each participant and with a larger sample size to generate more conclusive results for the classrooms observation portion of this study.
Previous work suggests that pedagogical knowledge of affective and metacognitive strategies (i.e. knowledge of students within TPKB) may be important for using studentcentered instructional approaches in higher education and conclude that "teaching professional development for active-learning instruction that does not help instructors plan for the cognitive, affective, and metacognitive dimensions of active learning will fall short of promoting effective instruction" (Auerbach and Andrews, 2018, p. 18). In the present study, the New Faculty Program components are intended to support new faculty in translating these dimensions into practice, particularly the affective and metacognitive dimensions. While we did not measure "effective instruction," nor did we capture evidence of student achievement, we may be able to speculate that those instructors who shifted to active learning have been doing so in a way that was effective for their students. Future work exploring the relationship between instructional practices and student outcomes is needed, particularly for programs such as the one in this study that attend to all dimensions of learning.

Institutional culture
Participants experienced significant increases to both dimensions of their sense of belonging to the institution with large practical effects. Participants increased their sense of connectedness after participating in CDI and maintained these levels throughout their FLC experience. For most participants, CDI marked the beginning of their cohort experience and with it their sense of being connected with members of the institution outside of their departments. The present study extends previous higher education research on learning JRIT communities (Kezar and Gehrke, 2017) to suggest that forming a cross-disciplinary FLC following a course design experience can impact new faculty's sense of being connected to a larger network at their institution. Our findings demonstrate that the FLC offered a continuation of a cohort experience and increased awareness of available continued support. Future research exploring engagement of these faculty in other learning communities is needed to understand the long-term impact of the program's FLC.
Conversely, after their CDI participation, there was no change to participants' beliefs that their continued pedagogical development was supported by the institution, but they did experience significant increases to these beliefs after their FLC experience. These increases may be attributed to our institutional context. University leadership is highly supportive of center work in general and of the New Faculty Program in particular, actively signaling that an investment in teaching is important for new faculty. The provost as well as most deans and department chairs invite and recommend new colleagues to the New Faculty Program. Research on the impact of institutional support for teaching suggests that low levels of support can be a large barrier for faculty in feeling connected to the university and in implementing learning-focused instruction (Shadle et al., 2017). The present study supports Henderson's finding that effective educational development interventions are designed in a way that is appropriate for a particular university system. Conversely, it may also suggest that small changes to how an institution signals the importance of teaching may have longranging impacts on faculty's sense of belonging. While not tested in the present study, institutional support may also explain improvements observed in other measured outcomes.

Assessing intervention impact
Our study addresses the shortcomings in prior assessment studies focused on limited data sources as a measure of intervention impact (Amundsen and Wilson, 2012;Cilliers and Herman, 2010). This study also addresses the call for research on university faculty to measure both beliefs and practices to obtain a complete picture of faculty and instruction in higher education (Kane et al., 2002). However, one limitation of the present study was the inability to connect individual faculty across data sources. Our future research aims to explore the impact of educational development interventions at the instructor level to understand the extent to which the intervention translates to understandings, practices and student outcomes.

Implications and future research
Our study demonstrates that educational development interventions designed by CTLs such as the New Faculty Program can be developed using both the empirical and theoretical literature. Through an intensive, sustained format (i.e. CDI combined with a semester-long learning community and individual teaching consultations), the New Faculty Program intentionally targeted multiple components of the TPK&S framework. However, we did not explicitly measure TPKB or TSPK. While some measures exist to assess TSPK components (Gess-Newsome et al., 2019;Seung et al., 2012), they have not been applied broadly in higher education, nor have they been coupled with assessments of TPKB or effective educational development interventions. Our work sets the foundation for integrating a theoretical framework into educational development interventions. Future research should explore the relationship between TPKB, TSPK, amplifiers and filters, and instructional practice in higher education.
Because participation in the New Faculty Program and its assessment is voluntary, there were several limitations to this study. First, due to a selection bias, the New Faculty Program faculty were likely more receptive to changing their teaching practices because they chose to participate in a program aimed at pedagogical development. Other current research in our center seeks to explore differences in faculty instructional practices and student success for Measuring institutional transformation the New Faculty Program faculty compared to faculty who have not engaged in our center (Wheeler and Bach, 2021). Second, voluntary participation in the program's assessment contributed to the differential sample sizes for the study's individual measures. Faculty may have experienced some fatigue in their participation persistence due to the high frequency of data collection inherent in our study design. The limited sample size prevented us from making inferences from observational data or determining a correlation between the endorsements of learning-focused course design principles and the use of evidence-based teaching practices. However, consistent with Kreber and Brook's (2001) caution, the intent of this study was not to make inferences to other contexts based on our results but rather provide insight into the ability of the New Faculty Program to meet the needs of participants and their students in our particular institutional culture. Finally, the second author of this study was in charge of program implementation, and there may have been some bias in reporting during data collection. Although a research team collected and anonymized the data, there may have been perceptions of being evaluated, particularly in the classroom observation phase of the study.
Future research should consider designs which maximize faculty participation in measurement across all data sources. This could include, among others, strategies for incentivizing participation in all assessments and for allaying concerns about being evaluated. Larger sample sizes and more sensitive observation protocols would allow for correlational studies to determine how changes in faculty beliefs translate into changes in teaching practice. The present study provides the groundwork for addressing calls for future research using approaches such as social network analysis (SNA) (Kezar, 2014) to understand the role CTLs play in higher education change.

Conclusion
In conclusion, our study has several practical applications for institutions, educational development practitioners and researchers. Institutions are well advised to heed the call to support research-based faculty development interventions and invest resources in sustained programming. Furthermore, intensive course design experiences followed by an FLC with targeted assignments and reflection can support faculty in translating TPKB and TSPK into practice. Institutions, and particularly those in key leadership positions at institutions, should actively signal that investment in teaching is important for new faculty. Finally, comprehensive and systematic research on educational development interventions is labor intensive and presents challenges for CTLs with limited resources. When possible, CTLs should look for opportunities to collaborate with educational research units on campus and align their assessment with institutional assessment plans to make use of existing resources. They might also leverage institutional data to illustrate the impact of their program. Notes 1. Because of the center's preexisting relationships with STEM departments through earlier programming, disciplinary overrepresentation of STEM faculty was expected. Although the program was open to faculty new to the institution regardless of rank, the overrepresentation of assistant professors in our sample may be explained through the larger interest of novice teachers in participating in intensive faculty development programs, while more senior hires may perceive that they possess relevant skills (Honey et al., 2014).
2. The classroom community scale wording was revised to reference the specific community of the university where participants were employed and also highlights the negative wording of reverse coded items.
3. Training included videotaped practice observations, live observations and feedback from the trainer. All observers achieved a Cohen's kappa (κ), >0.80 with a master observer, which demonstrated a strong to almost perfect level of agreement (McHugh, 2012). JRIT