(How) Does 360-degree feedback benefit the field of entrepreneurship?

Abstract

Purpose

Most of the studies in entrepreneurship depend on single-source rating methods to collect data on both predictors and criteria. The threat to effect sizes as a result of using single-source ratings is particularly relevant to psychology-based entrepreneurship research. Therefore, the purpose of this paper is to explore the prospects of applying 360-degree feedback to the field of entrepreneurship and to discuss a set of cases regarding how 360-degree feedback may boost effect sizes in entrepreneurship research.

Design/methodology/approach

A qualitative review of current literature was performed.

Findings

The review indicated that the effect sizes in psychology-based entrepreneurship research are mostly small and the use of single-source ratings is prevalent; some preliminary findings supported the utility of 360-degree feedback in entrepreneurship research; entrepreneurial orientation (EO) research may benefit from 360-degree feedback; and members of top management teams, employees from research and product development, sales agents, retail buying agents, store sales clerks, and consumers are all valid informants to provide ratings of EO.

Originality/value

The present study provided theoretical explanations and used empirical evidence to elucidate how 360-degree feedback may benefit the field of entrepreneurship. In addition, recommendations for future research using 360-degree feedback in entrepreneurship research were offered and discussed. A sample research study on EO using 360-degree feedback was delineated.

Citation

Miao, C., Humphrey, R.H., Qian, S. and Oh, I.-S. (2018), "(How) Does 360-degree feedback benefit the field of entrepreneurship?", New England Journal of Entrepreneurship, Vol. 21 No. 1, pp. 65-72. https://doi.org/10.1108/NEJE-04-2018-0009

The 360-degree feedback has existed for a long time, and it has become an important topic in the fields of industrial and organizational psychology and human resource management (Bracken et al., 2016). For example, 360-degree feedback has been recommended by researchers and practitioners to measure performance because it allows a much more comprehensive assessment of performance than single-source rating alone (Oh and Berry, 2009). Recently, Bracken et al. (2016) presented a review article to discuss the controversies in the 360-degree feedback literature. Yet, their review was dominated by examples from the fields of human resource management and industrial and organizational psychology. The present paper extends their recommendations to the field of entrepreneurship by providing examples of how 360-degree feedback may help to advance the understanding of important phenomena in the field of entrepreneurship.

Theoretical backgrounds of 360-degree feedback

Socio-analytic theory posits that self-reports and observer-reports have notable differences; the former evaluates the internal dynamics (e.g. identity) of an individual, whereas the latter assesses an individual’s reputation (Hogan, 1991; Oh et al., 2011). Reputation largely depends on one’s past performance and prior performance predicts one’s future performance in a similar context. Hence, one’s reputation is more predictive of actual behaviors than is one’s internal dynamics, because the aim of observer ratings is behavioral prediction (Oh et al., 2011). In sum, socio-analytic theory yields two important implications. First, using observer ratings to capture one’s social reputation or public self may be most appropriate for the goal of prediction; this is particularly true if observers closely interact with individuals who are assessed (Mount et al., 1994). Second, if self-report ratings and observer ratings capture different aspects of the individuals being assessed, then the combination of these two ratings should produce higher validity than either one of these rating methods used alone. This suggests that using multisource ratings (e.g. 360-degree feedback) should result in validity gains. Oh et al.’s (2011) meta-analytic findings provide support for these arguments. Except for emotional stability, Oh et al. found that operational validities of observer ratings of Big Five personality traits were substantially higher than those of self-ratings of Big Five personality traits, and the validities of the combination of self-ratings and observer ratings of Big Five personality traits were higher than ratings of either source. Since 360-degree feedback involves data collection from multiple different raters, Oh et al.’s meta-analytic study yields support for using 360-degree ratings to boost validities.

The above meta-analytic evidence demonstrates how the use of 360-degree ratings of predictors results in validity gain. Research evidence also shows validity gains as a result of the use of 360-degree ratings of a criterion. Since each rating source represents a unique and potentially valid perspective on a criterion, 360-degree ratings can address the problem of criterion deficiency because their use taps a greater fraction of a criterion than does any single-source rating (Oh and Berry, 2009). For example, Oh and Berry (2009) found that the operational validities of personality traits were increased by 50 to 74 percent when 360-degree ratings of the criterion were used compared to when single-source ratings of the criterion were used. In the following sections, we discuss the promise of using 360-degree feedback in the field of entrepreneurship.

Extensions of 360-degree feedback to the field of entrepreneurship

A preponderance of the entrepreneurship literature relies on single-source ratings (entrepreneurs’ self-reported ratings); as such, some relationships of interest may be underestimated due to the problems of response distortion, like self-enhancement (i.e. intentional faking), self-deception (unintentional response distortion), and the inability to capture “blind spots” (aspects of the target person that only others can see). The influence of self-reported ratings may be particularly relevant to psychology-based entrepreneurship research (e.g. Ahmetoglu et al., 2011; Akhtar et al., 2013; Brandstätter, 2011; Frese and Gielnik, 2014), an area of inquiry that uses psychology-based constructs (e.g. psychological traits and motivation) to predict entrepreneurial outcomes (Frese and Gielnik, 2014). In this research area, single-source self-ratings of predictors and criterion variables are ubiquitous.

Some meta-analytic evidence has corroborated the importance and relevance of this area of inquiry. For example, meta-analytic findings demonstrated that all Big Five personality traits except agreeableness are significant predictors of entrepreneurial intention, business creation, and entrepreneurial performance (Zhao and Seibert, 2006; Zhao et al., 2010). Rauch and Frese’s (2007) meta-analysis showed the significant predictive validity of a set of personality traits such as self-efficacy, achievement motivation, risk propensity, innovativeness, stress tolerance, autonomy, and locus of control in predicting business creation and entrepreneurial success.

However, a perusal of the magnitude of validity coefficients from these meta-analytic studies indicated that very few of them are on a par with Cohen’s (1988) benchmark of moderate validity (i.e. 0.30). Some validity coefficients, such as risk-taking, are only around 0.10, which is barely in line with the benchmark of small validity. Our observation is consistent with Brandstätter (2011), who concluded that “the effect sizes are mostly small” (p. 222). We suspect that over-reliance on using single-source self-ratings of these psychological traits may result in underestimation of validity coefficients and may mask the truth that some psychological traits that were deemed as non-significant predictors based on the studies using single-source self-ratings may be significant predictors of entrepreneurial outcomes when 360-degree feedback is employed. Based on socio-analytic theory and Oh et al.’s (2011) meta-analytic findings, we argue that using 360-degree rating methods may boost validity coefficients, and such validity gain may further substantiate the legitimacy of psychology-based entrepreneurship research and clarify the full picture of this area of inquiry.

Empirical evidence of 360-degree feedback’s utility in entrepreneurship research

The preliminary research that has been done supports the utility of 360-degree feedback in entrepreneurship research. For example, Miao et al. (e.g. Miao, 2015; Miao and Coombs, 2015) examined how psychological traits predict individuals’ intentions to create businesses and to take over businesses. They assessed four psychological traits, which are risk propensity, emotional intelligence, proactive personality, and rebelliousness. Three individuals who were familiar with each focal subject were invited to provide observer ratings of these four psychological traits. They performed both regression analyses and relative weights analyses. Their study made two noteworthy contributions that substantiated the importance of 360-degree feedback in entrepreneurship research.

First, observer ratings of all of the aforementioned four psychological traits demonstrated incremental validities in predicting entrepreneurial start-up intention above and beyond self-ratings of them. Observer ratings of risk propensity and rebelliousness showed incremental validities in predicting entrepreneurial take-over intention over and above self-report ratings of them.

Second, it appears that using observer ratings is more likely to contribute incremental validities above and beyond self-ratings of them when measuring socially undesirable traits. This is because individuals engaged in more response distortion when socially undesirable traits (e.g. rebelliousness and risk propensity) were assessed via self-report ratings (Miao, 2015; Miao and Coombs, 2015). Therefore, use of observer ratings may help to mitigate social desirability bias in entrepreneurship research.

The above preliminary empirical findings support the theory that multisource ratings may boost validities in entrepreneurship research. In the following section, we discuss how 360-degree feedback may be applied in future entrepreneurship research.

Future directions of using 360-degree feedback in entrepreneurship research

The prior empirical example supports the utility of 360-degree feedback in individual-level entrepreneurship research. We also believe that 360-degree feedback may benefit the research on firm level constructs in entrepreneurship, such as entrepreneurial orientation (EO) – a construct that consists of three salient dimensions (i.e. innovativeness, risk-taking, and proactiveness) and is construed as a top manager’s perception of a firm’s strategic stance (Frese and Gielnik, 2014; Miao et al., 2017; Rauch et al., 2009). Although EO is conceptualized as a firm level variable, it is actually measured at the individual level because it consists of the top manager’s individual psychological perception of EO. As such, it is susceptible to the powerful self-deception and impression management effects discussed previously. Top managers are likely to have strong motivations to see themselves, and their firms, as more innovative than they truly are. Even when top managers accurately report their own level of EO, they may have blind spots with regard to the EO of other employees scattered throughout the organization. In order to make EO an accurate firm level variable, EO should be measured throughout the organization at multiple levels and across multiple departments. This would result in a firm level measurement instead of an individual-level measurement of EO.

We argue that it may be useful to introduce additional raters of EO, such as customers, competitors, contractors, investors, or any other raters (stakeholders) who have direct, significant business-related interactions with the ratees (entrepreneurs) (Bracken et al., 2016). We propose that relevant other raters are in a better position to rate EO than are top managers, and the inclusion of their ratings may enhance validity coefficients. For example, customers may more accurately assess a firm’s innovativeness because they are the end-users of the firm’s products/services. A firm’s competitors may more accurately evaluate a firm’s proactiveness because they scrutinize the competing firm’s new products/services. Thus, incorporating 360-degree feedback may result in validity gain between EO and criterion variables because additional sources of ratings may capture unique variance that is not captured by single-source top managers’ ratings.

In sum, using multisource ratings of EO may serve two important purposes. First, averaging the scores from different rating sources of EO and employing an averaged value of EO will enhance the reliability and validity of EO (Mount et al., 1994). Second, the scores from each individual rating source can also be treated independently if one’s goal is to increase diagnostic validity, because different rating sources provide different perspectives of EO (Mount et al., 1994). For example, if one’s objective is to analyze the level of EO for a given firm and to use the information to improve a firm’s EO, then one may consider the scores of EO from each rating source in order to obtain a finer-grained view of a firm’s EO from different perspectives.

A sample research study on EO using 360-degree feedback

In light of the fact that there is no existing research, to the best of our knowledge, that used 360-degree feedback in EO research, we aim to delineate a sample study about how this can be done. The EO measure developed and validated by Covin and Slevin (1989) is the most widely used EO scale. This scale captures three major components of EO, which are innovativeness, risk-taking, and proactiveness. A sample study may capture these three components of EO by surveying different groups of informants and comparing and combining their responses on each item. When designing 360 degree and multi-rater studies, careful consideration should be given to the selection of the groups to be surveyed. As previously discussed, 360 degree and other multi-rater studies take advantage of the different yet valid knowledge and perspectives that different raters have. Since EO studies are concerned with innovativeness, risk-taking, and proactiveness, it is important to select survey groups who could be reasonably expected to have some knowledge of the firm’s performance along at least one of these dimensions, and to have differing perspectives as well.

In most EO studies, the entrepreneur, chief executive officer, and top management team are surveyed. Because of their positions at the top of the organization, they should have inside knowledge of the organization’s strategies and performance. Although the entrepreneur might be the best person to survey, other members of management and the top leadership team would also be good informants, and a multi-rating approach might reveal significant differences of opinion at this level. Thus, almost all multi-rating studies on EO should include the entrepreneur and members of the top management team, and these respondents are likely to have the broadest range of knowledge across the three dimensions of innovativeness, risk-taking, and proactiveness. Because of their knowledge of the firm’s finances and the cost of developing new products, they might be in the best position in particular to judge risk-taking.

The members of the top management team might also be good choices to provide ratings of the entrepreneur’s personality traits. Because they work closely with the entrepreneur, they would be in a good position to observe the entrepreneur’s personal characteristics. Thus, for studies that are examining the relationships between EO and the personality traits, values, or other characteristics of the entrepreneur, the other members of the top management team would be good choices to provide independent ratings of the entrepreneurs’ characteristics. These could be compared to the entrepreneurs’ self-ratings and researchers could test if their ratings provide incremental predictability for the personal characteristics → EO relationship.

In most cases, it might be useful to survey employees as well. Employees in product development might be in an especially good position to assess innovativeness because that is closely related to their jobs. In addition, they might know if the entrepreneur and the top management team are proactively following up on opportunities to develop new products. The organization’s sales agents might also be in a good position to judge innovation and proactiveness because they come directly into contact with customers (or retail buying agents) and they have to discuss how their products compare to those of their competitors.

People outside the organization may also have some insight into the organization’s EO. Retail buying agents, who make decisions about buying and stocking the organization’s products, may be in an excellent position to judge the organization’s EO. In many cases, they make buying decisions after comparing the product features with competitors’ products, and they should also know how well the products sell. Depending on the product, retail outlets may also have sales staff who explain products to customers and who are supposed to be knowledgeable about the products being sold and their relative features. These retail employees may be considerably better informed than the average consumer about products and may know which brands first introduced new and innovative features.

Consumers need to be surveyed because they are the end-users of firms’ products/services and their perspectives on the products offered are likely to differ considerably from the entrepreneurs and their employees. In addition to the normal ego-enhancing biases that the firm’s employees may have regarding their products, the product designers and other professional employees may regard innovation in terms of highly technical details, whereas consumers may regard innovation primarily in terms of ease of use and overall functionality. Depending on the product, many consumers do considerable product comparisons before buying a product, and they are in the best position to judge whether the product meets their needs. It might be useful to survey two groups of consumers: those who bought the product and those who ultimately decided to buy another product instead.

Any study of EO should include a variety of objective data, including sales growth, money spent on research and development, and similar data. These would normally be collected from the entrepreneur granting access to the researchers.

After these scores are collected from different sources, factor analyses should be performed to analyze the factorial structure of the measurement items of EO. If all of the measurement items load on a single factor, then the scores from measurement items can be combined into a single one (Stam and Elfring, 2008). Since the scores on the surveys were collected from different rating sources of EO, such a combination should result in higher reliability and higher validity of an EO scale. In addition, if one’s major objective is to boost diagnostic validity, then the scores from each rating source may be treated independently, because that would allow an accurate assessment of a firm’s level of EO across each component of EO, and consequently proper actions can be taken to improve a firm’s EO.

As the above considerations make clear, the respondents included in any multi-rater study on EO should have some knowledge about the innovativeness of the organizations’ products or services. For example, a study of entrepreneurial firms that sell electronic or similar goods through retail outlets might include the following groups in order to rate the organization’s EO:

1. The entrepreneur and at least two members of the top management team. In addition to rating EO, the entrepreneur and the members of the top management team could also provide ratings of the entrepreneur’s personality traits, values, or other characteristics.

2. Employees from research and product development.

3. The organization’s sales agents.

4. Retail buying agents who make decisions about buying and stocking the organization’s products.

5. Store sales clerks responsible for explaining and selling the product to consumers.

6. Consumers – both those who purchased the product and those who preferred to buy a competitor’s product instead.

7. Objective data on research and product development expenditures, sales, growth, etc.

These groups would comprise an ideal study, but clearly, not all studies would need to include all groups. Including these different groups would allow researchers to test if the use of 360-degree ratings of predictors results in validity gain for entrepreneurship research in the same way that they have improved the validity of personality measures in human resource management research.

Conclusion

The present paper discusses the prospects of applying 360-degree feedback to the field of entrepreneurship and calls for more frequent use of 360-degree feedback in this field. We encourage future entrepreneurship researchers to consider using 360-degree feedback to improve the criterion-related validities of entrepreneurship-related constructs. In many cases, using 360-degree feedback could increase the size of the effects being studied, and perhaps even turn non-significant findings into significant ones. This increase in effect sizes would be consistent with the human resources research on personality traits, which saw an increase in operational validity of up to 74 percent when using 360-degree ratings (Oh and Berry, 2009). The limited research in entrepreneurship using 360-degree ratings supports the view that their use increases validities. For example, Miao et al (Miao, 2015; Miao and Coombs, 2015) found that observer ratings of traits increased incremental validities over self-ratings when predicting individuals’ intentions to create businesses and to take over businesses.

Perhaps just as importantly, the use of 360-degree ratings obtained from employees throughout an organization creates true firm level measurements. Although EO is conceptualized as a firm level variable, it has traditionally been measured at the individual level. The 360-degree ratings can better assess the extent to which EO pervades an organization’s culture and is valued across all levels and divisions of the organization.

Although there are likely to be substantial benefits to using 360-degree ratings, it must be acknowledged that there are considerable costs and inconveniences involved in using them. A far greater number of respondents would have to be involved, which would be costly in terms of the employees’ time and effort. This cost could reduce the number of organizations willing to participate. The data analysis would also be substantially more complicated. On the other hand, the richness of the data gathered would be considerably enhanced, and the gathered data could be of considerably greater use to both the organization and researchers. For example, the 360-degree ratings could let top managers know the extent to which entrepreneurial attitudes are spread throughout the organization. Likewise, 360-degree ratings could inform them as to how their products are viewed by customers and buying agents.

Because of the complexities of doing 360-degree research, we do not feel that every study, or even most studies, should have to use 360-degree methods. However, the major findings in the field should be replicated using 360-degree methods in order to better establish the true effect sizes and relative importance of various variables. Journal editors and reviewers should recognize the value of such replications and encourage this type of research. In addition, researchers should use 360-degree ratings whenever there are likely to be problems of self-enhancement, self-deception, and an inability to capture blind spots. Because top managers are likely to overestimate their EO, it is important that at least some studies verify EO findings by using 360-degree ratings.