Improving entrepreneurial competencies in the classroom: an extension and in-study replication

Purpose – ArecentpaperbyMorris etal. (2013b)presentedevidencethatstudentscandevelopentrepreneurial competenciesthroughinternationalfieldwork.Thispaperexploreswhetherthesameresultscanbedevelopedinatraditionalclassroomsetting. Design/methodology/approach – The study is a systematic replication of the Morris study with the addition of a matched pair, quasi-experimental design, with a self-replication. Data were collected on 13 self-reported competencies at the start of a semester from two groups using the Morris instrument. The treatment group was exposed to a curriculum designed to teach entrepreneurial competencies, and both groups were re-surveyed at the end of the semester. The process was then repeated with a different cohort, one year later, to replicate the initial study. Findings – Five competencies saw significant increases in the first treatment group. However, only three of thesecompetenciesincreasedmoreinthetreatmentgroupthanthecontrolgroup.Inthereplicationstudy,onlyonecompetencywassignificantlyhigherinthetreatmentgroup,andthatcompetencywasnotoneoftheoriginalthree. Practical implications – Educators and policymakers should select a curriculum that is valid and reliable. Entrepreneurship educators and policymaker should devote more time to evaluating the effectiveness of different pedagogical techniques for improving entrepreneurial competencies. Originality/value – Tothebestofourknowledge,thisisoneofthefirststudiesinentrepreneurshipeducation to undertake a matched pair, quasi-experimental design with an in-study replication. The results indicate that serious inferential errors arise if simpler designs are used, even though such designs are the norm in entrepreneurship research.


Introduction
In 2013, Michael Morris and his colleagues reported in the Journal of Small Business Management that 40 students reported a significant increase in 9 of 13 entrepreneurial competencies following a six-week international consulting project with disadvantaged entrepreneurs in South Africa (Morris et al., 2013b). At the time, the paper was described as a pilot study, but the results have assumed greater importance given the rise of competencybased education in the interim.
A competency refers to the "knowledge, skills, attitudes, values, and behaviors that people need to successfully perform a particular task or activity" (Morris et al., 2013a, p. 45).
Competence, the state of being proficient in a competency, has been associated with higher performance and productivity. There is also a widespread belief that competence can be developed over time with training and practice (Morris et al., 2013a).
Competency-based education is defined as "an outcome-based approach to education that incorporates modes of instructional delivery and assessment efforts designed to evaluate mastery of learning by students through their demonstration of the knowledge, attitudes, values, skills, and behaviors required for the degree sought" (Gervais, 2016, p. 99). In recent years, there has been a renewed interest in competence-based education, with educational institutions like Western Governor's University and Southern New Hampshire University making the demonstration of competence via project-based learning the centerpiece of their pedagogical strategy (Adams, 2019). Enrollments at both schools regularly exceed 100,000 students.
Presumably, competence-based entrepreneurship education (CBEE) requires the acquisition of knowledge, skills, attitudes, values and behaviors which result in ". . . venture birth, survival, and/or growth" (Bird, 1995, p. 51). However, Morris et al. (2013a) have argued "there is little consensus in regarding the relative importance of particular competencies in an entrepreneurial context" (p. 46). The finding by Morris et al. (2013b) that a planned intervention can have a measurable impact on entrepreneurial competencies in a short period of time is therefore of great importance for entrepreneurship educators and competence-based education more generally.
This paper extends the seminal work of Morris et al. (2013b) by asking whether perceived entrepreneurial competencies can be improved in a semester-long classroom setting. Few students can afford to spend six weeks abroad on an international consulting project to improve their entrepreneurial competencies, so the question is of intrinsic interest to educators serving lower socio-economic populations. While this study is not a direct replication of the initial paper, it is a form of systematic replication using the same instrument. Systematic replication "establishes the generality of a phenomenon over a wide range of situations, [including] varied intervention procedures, participant characteristics, implementers, and settings" (Tincani and Travers, 2019, p. 61).

The original study
The original study (Morris et al., 2013b) was conducted in two phases. First, it engaged 10 experienced entrepreneurs and 10 entrepreneurship faculty in a Delphi study to identify a set of entrepreneurial competencies. Morris et al. were able to move from an initial list of 265 competencies to a consensus list of 13 competencies specific to entrepreneurship, namely: opportunity recognition, opportunity assessment, risk management/mitigation, conveying a compelling vision, tenacity/perseverance, creative problem solving/imaginativeness, resource leveraging, guerilla skills, value creation, maintain focus yet adapt, resilience, self-efficacy and building and using networks.
Once the list of entrepreneurial competencies was established, the authors sought measurement instruments for each competency from existing tested instruments that involved self-reporting. The authors created a 131-item instrument by selecting 112 items from 11 prior research instruments and adding 19 items of their own for risk management/ mitigation (9), conveying a compelling vision (8) and guerilla skills (2). The instrument was designed for self-reporting and the items from prior research all had acceptable reliability.
Finally, the authors applied this instrument to a convenience sample of students engaged in an intensive international consulting experience to assess whether entrepreneurial competencies could be improved through experiential entrepreneurship education. The authors did not create specific hypotheses for each competency; simply being curious about whether any of the self-reported competencies could be improved during the experience. The student sample consisted of 40 mixed undergraduate and graduate students within business and non-business programs with 25 students from the United States and 15 students from South Africa. The entrepreneurship education experience included three months of online instruction and six weeks of half-day classroom instruction plus field work in teams consulting with South African micro entrepreneurs for six weeks. Entrepreneurial competencies were not taught explicitly. Students completed the self-reporting using the Morris Instrument before and after the experience.
Ultimately, Morris et al. found that self-reported entrepreneurial competencies did increase after the entrepreneurship education. Using 78 of the original 131 items in the final analysis, they found improvements in all 13 competencies with the differences in mean scores between pre-and post-test scores significant for 9 competencies. According to the authors, "These results lend initial support to the hypothesis that competencies can be enhanced based on exposure to an entrepreneurship program, in this case one that was centered both on the classroom and intense experiential learning. When placed in a context that centers on experiencing and performing real tasks that support entrepreneurial outcomes, students appear to get better at demonstrating specific competencies" (Morris et al., 2013b, p. 361).
Based on the results, the authors recommend increased emphasis on experiential education in entrepreneurship, including pedagogies such as entrepreneur interviews, developing business models or plans, feasibility analyses, and/or "elevator pitches, business plan presentations, or construct mock entrepreneurial tasks or simulations" (Morris et al., 2013b, p. 363). They further state, "[W]e believe our research presents a foundation for guiding the development of entrepreneurship curricula. Courses can be tailored to developing specific competencies within students. . ." (Morris et al., 2013b, p. 364). While Morris et al. seem to claim generalizability, they caution that "We do not expect that all experiential learning tasks will lead to development of all competencies" (Morris et al., 2013b, p. 365).

The current study
The impetus for the present research was to explicitly foster an understanding of and growth in entrepreneurial competencies among early academic career students by emphasizing the 13 core competencies identified and measured by Morris et al. (2013b). The Morris study was the most comprehensive model available and, as indicated above, provided insights into the types of experiential education that should be offered. The results were nothing short of remarkable, showing improvements in competencies for the relatively small sample of 40 students across grades, disciplines and nationalities, without direct instruction in the entrepreneurial competencies. This led to an interest in determining how and whether such results could be replicated in a more common collegiate environment, namely an on-campus, full semester undergraduate course.
We sought to enhance the methods of the original study through a systematic/conceptual replication rather than a reproduction, recognizing that there is a continuum of replication options (Walker et al., 2018). We retained the same measurement instrument as Morris et al. and the pre-test and post-test design. However, we were concerned that the small sample size in the original study made it difficult to discern small true effects. As a result, we opted to use matched pairs in our design, which gave us approximately 10 times the statistical power of the original study (Hollenbeck and Wright, 2017). This is akin to using a larger telescope to search for faint objects in the night sky (Simonsohn, 2015).
In addition, our sample selection was deliberate and well-defined. By selecting a group of students enrolled in an initial college entrepreneurship course within the university core curriculum we were able to control for heterogeneity and create a larger sample than in the original study. With little previous exposure to entrepreneurship education, these students Improving entrepreneurial competencies were poised to make the largest gains in self-reported competencies. The opportunity to add a control group of students with similar attributes also added validity to our results by ensuring that any gains were not just due to the passage of time or the general college experience but specific to the intervention. Similarly, we replicated the original design twelve months later to ensure that the findings could be reliability reproduced. It is interesting that Morris et al. did not posit any specific hypotheses about which individual competencies were expected to increase because of the consulting experience. We chose to not do so in our study either. While the allure of running many significance tests and HARKing (Hypothesizing After Results are Known) is strong per Hollenbeck and Wright (2017), we resist the development of post-hoc generated hypotheses to provide affirmative results.
This study follows the prescription regarding replication which suggests that, "Replication needs to become a normal scientific practice. . .to improve the quality of scholarship, accumulate knowledge, develop midrange theories, distill lessons for practice, and identify robust size effects and boundary conditions" (Walker et al., 2018, p. 623). With a high-powered systematic replication like ours, with interventions specifically targeting the 13 competencies, we fully expected to find similar results to Morris et al. (2013b). Any failure to find a reliable association provides a check on the generalizability of the Morris et al. results. It is important for the field to report whether a provocative finding is supported or unsupported in subsequent tests. Doing so avoids the "file drawer" effect of publication bias, confirmation bias and impacts of meta-analysis by reporting the results are they arose without HARKing. Walker et al. (2018, p. 623) make a valid point when they state, "Replication failures are not necessarily failures of good scientific practice; rather, they should be seen as the scientific process in action."

Literature review
Developmental and experiential pedagogies have long been the tools of entrepreneurship educators in their efforts to impact entrepreneurial activities (Albornoz, 2011). Yet, there have been ongoing changes to the pedagogical questions of what to teach and how to teach it as educators continue to learn more about how students acquire, retain and apply knowledge and skills and, consequently, which pedagogies are most effective in higher education (Neck and Greene, 2011;Rubin et al., 2018). To answer these questions, the goals and objectives of entrepreneurship education must be considered. Since its infancy, one clear purpose of entrepreneurship education has been to teach students how to establish new ventures (Rasmussen and Sorheim, 2006). Even with this singular goal, inquiries continue regarding the best practices in terms of organization and content (Neck and Greene, 2011;Ronstadt, 1987;Rubin et al., 2018). Neck and Green (2011) report that gaining consensus about the essential and most valuable learning outcomes from entrepreneurship education continues to be elusive, although fostering an entrepreneurial mindset (McGrath and MacMillan, 2000) and developing entrepreneurial competencies have generally emerged as leading outcomes.
One way of understanding what works is to use evidence-based decision-making, and to develop and test assessments of mastery through scientific methods (including replication). Evidence-based decision-making means that decisions should be based on the latest and best evidence of what works. Evidence-based techniques began in medicine but have since been extended to management (Pfeffer and Sutton, 2006) and education (Davies, 1999;Slavin, 2002). Applying this concept to entrepreneurship education means sorting through research findings and identifying what constitutes good evidence. However, such information continues to be lacking in the field, and research has indicated a gap between the startup activities of nascent entrepreneurs and what is taught in entrepreneurship textbooks (Edelman et al., 2008). One of the few long-term studies in this area found that the number of entrepreneurship courses taken did not correlate with the operational performance of new ventures and that writing a business plan had no effect on success beyond initial fund raising (Lange et al., 2012).
Over time, the emerging approaches to entrepreneurial education have evolved into a distinction between a competency-based approach and a personality-based approach. Kyndt and Baert (2015, p. 14) have argued that entrepreneurial competencies as defined by Bird (1995) are "changeable, learnable and attainable through experience, training, or coaching" and highlight differences between the competency-based approach and a personality-based approach that presumes characteristics are comparatively immutable.
One of the appeals of the competency-based approach is that it supports the proposition that entrepreneurship can be taught (Dickson et al., 2008;Fayolle et al., 2005;Kuratko, 2005). It also allows entrepreneurship educators to argue that entrepreneurial competencies (such as innovativeness, proactiveness and risk-taking) are distinct from general business competencies and thus can be taught outside a business school context (Chandler and Jansen, 1992;Morris and Kaplan, 2014;Rubin et al., 2018). It also addresses the implicit long chain of causal reasoning that links competency training with entrepreneurial success as depicted in Figure 1 (Glackin et al., 2018). Demonstrating that entrepreneurial competencies can be clearly defined, learned and applied to affect entrepreneurial outcomes is therefore vital to the successful adaptation of the competency-based approach to entrepreneurship.

Competency frameworks
Although there has been some gradual convergence, there is still a lack of consensus on the definition of a canonical collection of entrepreneurial competencies, and some argue that this lack of consensus constrains progress in entrepreneurship education (Mitchelmore and Rowley, 2010). Prior frameworks have also been criticized by Morris et al. (2013b) due to their focus on too few competencies (often only four to six items) although research has continued to emphasize one or a few competencies rather than a comprehensive framework (Gonzalez-Lopez et al., 2018;Lee et al., 2018;Oosterbeek et al., 2010;Rezaei-Zadeh et al., 2017;S anchez, 2011).
The past seven years has seen a concerted effort to create entrepreneurial competency frameworks such as the European Commission's EntreComp Framework Academics have also created frameworks and tested them to varying degrees (Gonzalez-Lopez et al., 2018;Lee et al., 2018;Oosterbeek et al., 2010;Rezaei-Zadeh et al., 2017;S anchez, 2011).
The recognized value of entrepreneurship to national economic outcomes has led some governments to create comprehensive frameworks. Examples include reports from the UK government on enterprise education (Davies, 2002). The European Commission's EntreComp framework (Bacigalupo et al., 2016), and the U.S. Department of Labor's Entrepreneurship Competency Model (Consortium for Entrepreneurship Education, 2009). The EntreComp framework includes an extensive roster of entrepreneurial competencies and related learning outcomes (15 competencies and 442 learning outcomes) leading to certifications for secondary school students. Recently, the Evaluation of Entrepreneurship Programs in Higher Education Institutions and Centres (EEEPHEIC), supported by the European Commission and led by Ecorys, has begun to develop a toolkit and methodology for measuring the value of entrepreneurship education with an emphasis on impact (see http://epic.ecorys.com).
This paper employs the entrepreneurial competence framework created by Morris et al. Delphi survey of successful entrepreneurship educators and entrepreneurs providing it some face validity. Second, it is one of the most inclusive frameworks of competencies and intersects with the EntreComp framework in at least nine areas. Most significantly, it specifies a measurement tool for each competence and has published data for comparative purposes. Table 1 contains a list of the thirteen competencies, definitions and sample questions.

Measuring competence
Entrepreneurship researchers have used several approaches to measure entrepreneurial competencies. Self-reporting is a prevalent method, but less than ideal. For instance, Van de Mortel (2008) suggests that social desirability response bias is problematic in self-report research. Entrepreneurial competencies have overwhelmingly been measured using Likert scales in the past, with more recent studies transitioning to behavioral measures (Kyndt and Baert, 2015;Schelfhout et al., 2016). While a direct behavioral demonstration of a competence is probably the most desirable form of measurement, this must be weighed against the cost and time required to collect such data, including the development of activities and scoring systems that capture the relevant facets of the underlying competence. For these reasons, most competence frameworks use Likert scales to measure changes in competence albeit with the clear recognition that they will not correlate as well with the desired end behaviors nor serve as the best measure for a given competence (Ajzen, 2005). The current study also adopts this convention. Table 2 illustrates several studies and their methods. Of the fourteen studies in the table (not including our own), fully 100% rely on self-reporting of competency, but only 35% utilized a pre/post design, and only 42% utilized a control group. None of the previous studies attempted to replicate their initial results.
The results of these studies should be taken within the context of their methods and the rigor of the studies. While several studies have been conducted on the acquisition of entrepreneurial competencies, few are methodologically rigorous. Rauch and Hulsink (2015, p. 188) support the work of Fiet (2001) and Weaver et al. (2006) in stating, "In light of methodological weaknesses inherent in entrepreneurship education studies, some authors concluded that there is little evidence as to its effectiveness." Rauch and Hulsink recommend the use of quasi-experimental designs with pre-and post-tests, timed according to the nature of outcomes to be studied.
K€ ohler et al. (2017) take this notion a step further by suggesting that studies, should be designed to exclude alternate explanations for significant effects, including the use of repeated measures and randomized or quasi-randomized control groups. Unfortunately, fewer than 50% of studies they reviewed in the Academy of Management Learning and Education journal followed this practice and virtually no studies utilized control groups. Similarly, Longva and Foss (2018) performed a systematic literature review (SLR) of 613 articles from 65 Association of Business Schools (ABS) journals and found only 17 utilized an experimental or quasi-experimental design. They lamented that a growth in rigor has not matched the growth in significant results. Only research by Gielnik et al. (2015) and Karlsson and Moberg (2013) rise to the Longva and Foss rigor standards with respect to this topic.
Even reaching further, Simmons et al. (2011) suggest a disclosure-based solution to presenting anything as significant with guidelines and requirements for authors and reviewers, including disclosure of all variables considered, data collection procedures and decisions, failed manipulations, and replication. Minimizing false positives using in-study replications has been recommended (Hsu et al., 2017;Uncles and Kwok, 2013). We deliberately incorporate these recommendations by using a quasi-experimental design with matched pair, pre-and posttests and in-study replication. The following sections provide details regarding our methodology and findings.

Competency
Definition from Morris et al. (2013b) Sample question

Opportunity recognition
The capacity to perceive changed conditions or overlooked possibilities in the environment that represent potential sources of profit or return to a venture I am good at "connecting dots" Creative problem solving/ imaginativeness The ability to relate previously unrelated objects or variables to produce novel and appropriate or useful outcomes I think outside of the box

Conveying a compelling vision
The ability to conceive an image of a future organizational state and to articulate that image in a manner that empowers followers to enact it I believe in a bold and daring view of the future

Opportunity assessment
The ability to evaluate the content structure of opportunities to accurately determine their relative attractiveness I can distinguish between profitable opportunities and not so profitable opportunities Self-efficacy The ability to maintain a sense of selfconfidence regarding one's ability to accomplish a particular task or attain a level of performance  (Oosterbeek et al., 2010). Interestingly, Oosterbeek et al. (2010, p. 442) report regarding a leading entrepreneurship education for college students, the "Program does not have the intended effects: the effect on students' self-assessed entrepreneurial skills is insignificant and the effect on the intention to become an entrepreneur is even negative." In addition, with the assessment requirements from regional, national and international accrediting bodies, there is pressure to demonstrate successful student learning outcomes. Based on these factors, we expected a course structured around building entrepreneurial competencies to result in large effects. As a university core course under the "life skills" category at our institution, Entrepreneurial Thinking was designed to foster the development and growth of core entrepreneurial competencies in students across the university. The course was an attractive site to explore competency development as at the time the course was developed, entrepreneurial competencies described by Morris et al. (2013b) appeared to suit this course. Entrepreneurial Thinking was a twice weekly, 16-weeks semester course with sufficient time to cover each topic adequately, to incorporate numerous experiential exercises, and to engage students in understanding the concepts. Students were to identify entrepreneurial competencies in public figures and reflecting on past and future application in their own lives. It seemed logical to teach core competencies in the earliest entrepreneurship course and build on them in subsequent courses. An overview of entrepreneurship and the Morris core competencies kicked off the course, followed by a weekly focus on the individual competencies combining mini-lectures, experiential exercises, videos, guest speakers and reflective writing. By the end of the semester, we expected students graded as proficient in the course to be able to articulate and explain competencies, apply them to outside parties and in their own lives.

Design
This paper uses an innovative design for research in this area beyond the use of the Morris instrument. First, a matched pairs design is adopted. Previous studies have compared the mean of the treatment group before and after an intervention (Uncles and Kwok, 2013). A matched pairs design, on the other hand, compares the mean change in competency for each individual. This design increases statistical power by an order of magnitude therefore making the design more sensitive to changes in small samples. This approach addresses a frequent criticism that replications lack statistical power. Second, we utilize a control group of students from a similar class (Financial Literacy -FINC100). Several others have used control groups (Gonzalez-Lopez et al., 2018;Hosseini and Pouratashi, 2011;Oosterbeek et al., 2010;Rauch and Hulsink, 2015;S anchez, 2011;Solesvik, 2019). However, only a few studies use both pre-and posttests and control groups as we do (Gonzalez-Lopez et al., 2018;Oosterbeek et al., 2010;Rauch and Hulsink, 2015;S anchez, 2011). Arguably, any change in the dependent variable might have occurred without the planned intervention (for instance, due to the Hawthorne effect, self-selection, test familiarity, or simply the passage of time). An effect that is experienced by the treatment group but not the control group mitigates against such claims. In a true experiment, students would be randomly assigned to the treatment and control groups. We do the next best thing by using another 100-level business class in the life skills section of the general education curriculum. The data in Table 2 shows that the demographics of the two classes are very similar.
Third, we conduct an in-study replication. If taking Entrepreneurial Thinking really improves competencies and the results are robust, then we should have been able to replicate the results from semester to semester with the same significant relationships in the replications (assuming significant statistical power). This approach is recommended by Hsu et al. (2017) as a best practice for re-confirming the validity of a given relationship. Our replication occurred exactly one year from the first study and used the same instructor to deliver the course to ensure students received a common experience.

Sample
All students taking the freshman courses Entrepreneurial Thinking (ENTR100) and Financial Literacy (FINC100) at a mid-sized Historically Black University in the Southeastern United States were invited to participate in the survey. In the first cohort, completed in Fall 2016, a total of 89 students took the instrument with 57 students taking the instrument twice (30 ENTR, 27 FINC). In the second cohort, administered in Fall 2017, a total of 77 students took Improving entrepreneurial competencies the instrument at least once, with 53 students taking the instrument twice (30 ENTR, 23 FINC). Eligible students who did not take the survey were simply absent on the day of administration rather than refusing to participate in the study. Demographic information was collected from the 110 matched surveys that were completed across the two cohorts. This information is presented in Table 3. The entrepreneurship students were more likely to be male lower division students but did not differ markedly in age, number of previous entrepreneurship courses, or race.

Instrument
The study utilized the Morris Competency Instrument (MCI) introduced by Morris et al. (2013b). The instrument contains 104 items on a five-point Likert scale ranging from Strongly Agree to Strongly Disagree or Frequently to Never. Descriptive statistics for the MCI are presented in Table 3. Our results indicate low reliabilities for 6 of the 13 scales in the 104-item version whereas Morris reported scores above 0.6 on all scales. Attempts to replicate the factor structure of the subscales in Morris et al. (2013b) were unsuccessful.

Procedure
The pre-treatment instrument was administered to each class section during the second week of classes and the post-treatment surveys were completed during the final week of classes. All students who were present on the day of the assessment and over age 18 were invited to participate in the study.
During the semester, students in the Entrepreneurial Thinking (ENTR 100) class received specific training in each of the 13 Morris competencies through a combination of lectures, videos, role playing, guest speakers and classroom exercises. There were no prerequisites for this freshman-level core curriculum life skills class. Students were assigned readings and videos from popular media rather than having a textbook. Each week, they learned about one of the competencies with at least one experiential activity per week to reinforce the readings, videos and lectures. Guest speakers provided "live case studies" for the students. The program was developed by the primary author based upon recommendations in the Morris et al. , and in consultation with other entrepreneurship faculty at the university. Students were not required to engage in a consulting experience unlike Morris et al. (2013b). However, they invested considerable time in learning about each competency, viewing videos of and identifying people who exhibited the competency, having guest speakers and engaging in experiential exercises related to each competency. They also selected "Superhero" entrepreneurs and apply their understanding of competencies by identifying and articulating examples with their chosen entrepreneurs. Finally, students reflected on how they exhibited entrepreneurial competencies in their personal lives and how they could incorporate them into their future lives.

Results/findings
In our initial analysis, the dependent variables in Table 4 were entered into a multiple regression model with the demographic variables in Table 3 as controls. Table 5 depicts the base model with the average competency score for all items as the dependent variable using the data from all four collection periods. Students who had taken more than one entrepreneurship course rated themselves higher on entrepreneurial competence, but none of the other parameters were significantly different from zero in the base model, and the overall model was not significant (mean 5 3.54, n 5 268, R 2 5 0.045, F 5 1.75, p > 0.05).
The central hypothesis in the study was that the post-instruction competencies would be higher than the pre-instruction competencies for the entrepreneurship class but not the finance class. The change in competence for each class and cohort is presented in Table 6. In the first cohort, competencies increased in five areas and declined in one area in the entrepreneurship class, with an overall increase in average competency from the start of the semester to the end. However, the finance class also saw an increase in two areas and a decline in two areas. Overall, the predicted interaction effect occurred in three of the 13 competencies.
In the replication study, we would expect to see the same interaction patterns if the treatment is reliably increasing the identified competencies. However, the desired interaction effect was present for only one competency that was not one of the initial three significant In the final analysis, we created a repeated measures multivariate model using Student ID as a random effect, and then entering control variables and independent variables with separate terms for cohort, class, time (pre/post) and the interaction between class and time. The results of this analysis are presented in Table 7. The model overall had a good fit due to controlling for within-subject variance and using the full sample size rather than matched pairs (mean 5 3.54, n 5 268, R 2 5 0.88), but no interaction between class and time was detected. There was, however, a significant change in scores over time, with post-semester scores being significantly higher for both classes. Similar results occurred with each competency entered as the dependent variable. However, two competencies (risk mitigation and creative problem solving) showed an interaction effect in the expected direction with the larger sample size.

Discussion
This study was motivated by the desire to extend the work of Morris et al. (2013b) and to explore whether we could improve entrepreneurial competencies in a college classroom. The principal hypothesis in this study was that the 13 entrepreneurial competencies measured by the Morris Competency Instrument (MCI) could be systematically raised over the course of a semester through classroom instruction. The results from the current sample show only very limited support for this hypothesis. To start, half of the sub-scales were not reliable, including resource leveraging and risk mitigation, both significant factors in some analyses.
The results basically indicated that we were unable to reliably increase any of the 13 competencies in the classroom. One explanation for this result is that these competencies cannot be developed in the classroom. Another explanation is that the wrong pedagogical techniques were used. The solution to both issues is to keep experimenting with pedagogical techniques targeted at improving individual competencies. When positive results are found then these results should be archived and made available to teachers of entrepreneurship, thus promoting evidence-based entrepreneurship education (Frese et al., 2012). Over time, it might then be possible to build a full semester long course that draws on this evidence-based material.
The failure to replicate the significant interaction effects in the first cohort also alerted us to the possibility that we were not seeing true gains in reported competencies, but this result would not have been detected with a typical one-shot design. This lack of replication is indicative of other findings in the social sciences, collectively referred to as the "replication crisis" (Open Science Collaboration, 2015). Small sample sizes often have low statistical power making it difficult to discern small true effect, and a large number of significance tests can lead to the phenomenon of "HARKing," Hypothesizing After Results are Known (Hollenbeck  (2005) has stated that most published research findings are false because of these practices. On a positive note, perceived competence in creative problem solving and opportunity recognition increased significantly in both entrepreneurship cohorts. This suggests that the exercises around creative problem solving and opportunity recognition might be capable of producing reliable increases in perceived competence. In the case of creative problem solving, the exercise used was "connecting the dots" where students listed their passions, hobbies, and frustrations on index cards. The cards were then grouped in three piles and students randomly drew a passion, hobby and frustration and attempted to brainstorm a business from the combination. Students then reflected on the best and worst ideas (Glackin, 2016). The opportunity recognition section of the course used a similar exercise called Milliken's "exercise in ridiculosity" (Shields, 2015). This exercise follows a similar approach to the connected dots exercise by placing cards in three piles labeled resources, skills and situations. Students are then required to use the business model canvas to create a business with the combination selected and reflect on the results. We encourage entrepreneurship educators to consider incorporating these exercises into their courses to verify their effectiveness in raising perceived competencies in the designated areas.

Limitations and future research
One limitation of the study is that it is a quasi-experiment not a true experiment. Subjects were not randomly assigned to the treatment and control groups. This means there might be some unobserved heterogeneity between the groups that accounts for the observed increase in creative problem solving and opportunity recognition despite our finding that the groups did not differ on several demographic variables.
Another possible explanation for the pattern of results is that each wave did not receive the same treatment. This may have led to subtle changes in the curriculum that were manifested as different survey results. Similarly, the curriculum was designed by a third party so may not have been faithfully implemented by either instructor even if perfectly designed to elicit the desired results. It was also discovered that a monthly reflective assignment requirement was dropped from the second cohort, which may have resulted in a different experience. Controlling for treatment effects over a semester is difficult but could be mitigated through extensive training of the instructors prior to taking the course to ensure a consistent application of the curriculum.
There were also issues with the reliability of the measurement instrument. An examination of the Cronbach's alphas in Table 3 suggests a low level of reliability (under 0.70) for six of the 13 scales, whereas Morris et al. (2013b) reported alphas over 0.70 for all scales. This means the items in the scale are not measuring the same underlying construct in our sample and further work is needed to see if this is a product of the context of our study or a broader issue with the inventory. Discussions with the instructor after the class also suggested that the large gain in "guerilla skills" may be at least partially attributable to an increased comprehension of the word "guerilla" as much as any increase in perceived competency. This suggests that the inventory may have to be reworked for different comprehension levels. Future work should also focus on using a matched pair design to maximize statistical power and use in-study replications to minimize false positives (Hsu et al., 2017;Uncles and Kwok, 2013).

Conclusion
One reason for studying entrepreneurial competencies is to determine the critical knowledge, skills, and abilities for undergraduate entrepreneurship students to acquire during their degree programs. This represents a significant challenge for entrepreneurship educators. Students in undergraduate degree programs have approximately 300 contact hours in entrepreneurship courses (and perhaps 1,000 h in total if you include outside study, internships and work experience). Even counting other business-related courses, contact hours will rarely reach more than 1,000 h in a degree program. This leads to two observations. First, entrepreneurship students will probably not master the field during their undergraduate education (Ericsson et al., 1993). Second, entrepreneurship educators do not have time to waste teaching the wrong content or using ineffective pedagogies given the limited hours available. This highlights the importance of conducting quality research that assesses student learning across various entrepreneurship education pedagogies and tools and yield effective guidance for instructors to use their limited engagement time most effectively.
It is both valuable and important to continue to replicate our efforts with other groups, in other settings, and over time. The ability to raise perceived competencies is an important step in entrepreneurial education but perceived competency is not actual competence, and there is still a requirement to demonstrate that actual competence affects entrepreneurial intention, entrepreneurial founding and, ultimately, the performance of a new venture (Gonzalez-Lopez et al., 2018). Research relating the development of entrepreneurial competencies to intention, found and performance is a vital future step. This work forms an important first step in creating a body of evidence-based knowledge that can guide entrepreneurship education into the future.