Meta-analytic findings: updating research and supporting practice on workplace issues

Citation

Viswesvaran, C., Ones, D.S. and Mesmer-Magnus, J. (2017), "Meta-analytic findings: updating research and supporting practice on workplace issues", Career Development International, Vol. 22 No. 5, pp. 462-468. https://doi.org/10.1108/CDI-08-2017-0135

Meta-analytic findings: updating research and supporting practice on workplace issues

Dynamic and turbulent changes in the workplace have necessitated new approaches to individual career management. Protean and boundaryless careers have become the norm while the traditional careers where an individual stays in the same occupation, much less the same organization, for their entire work life have become rare (Sullivan and Baruch, 2009), at least in modern societies. The increased globalization and changing demographic nature of the workforce has also substantially altered the psychological contract between workers and organizations. Nowadays, individuals change their employers frequently, thereby injecting new dynamics into areas of teamwork, citizenship behaviors, and work-family balance (Briscoe and Hall, 2006). The meaning of career success has also expanded from objective measures such as promotions and salary to career satisfaction and the drive to find psychological meaning at work. Research has shown that when individuals are more engaged in work, performance and psychological health improves (Mesmer-Magnus and Viswesvaran, 2010; Thomas et al., 2010).

The literature in career management has also acknowledged these dynamic changes by exploring the motives and interests that drive new generations of employees. Although several empirical studies have considered the antecedents and consequences of these rapid dynamic changes in the workplace, as is usually the case in all areas of applied research, conflicting findings abound. Data come obscured by several statistical artifacts confounding the observed effect sizes and relationships of interest. Further, as the volume of studies increases, it is virtually impossible to narratively summarize the multiple studies in an objective manner. Systematic quantitative reviews such as meta-analysis are needed (Schmidt and Hunter, 2014).

Since their introduction in 1970s, formalized methods of psychometric meta-analysis (cf. Hedges and Olkin, 1985; Schmidt and Hunter, 2014; Wilson and Lipsey, 2001) have helped researchers to empirically, systematically, and objectively synthesize large bodies of empirical research. Meta-analysis refers to a set of statistical procedures designed to cumulate the results of primary research in order to allow the estimation of population-level relations among variables (Glass, 1977; Ones et al., 2017; Stone and Rosopa, 2017). Indeed, without systematic and quantitative meta-analytic methods, it would be impossible to conduct an unbiased statistical integration of effect sizes across the multitudes of studies that exist on a given topic (Schmidt and Hunter, 2014). It is well known that individual studies often produce conflicting findings due to the biasing effects of statistical artifacts such as sampling error, range restriction, and unreliability in measures, among others (Ones et al., 2017; Schmidt and Hunter, 2014). Such conflicting findings raise questions about the generalizability of the focal relationships. In their 2014 paper on the use of meta-analysis for the advancement of scientific knowledge, Viswesvaran, Ones, Schmidt, Le and Oh stated that the only solid path to cumulative knowledge within a discipline requires the use of psychometric meta-analysis. Early validity generalization studies (for an overview, see Schmidt, 2015) demonstrated the distorting effect that statistical artifacts have on individual study results (Ones et al., 2017) and that unless the primary study has a very large sample size, it is prone to sampling error and low power which can affect the results and potential conclusions drawn (Stone and Rosopa, 2017). Psychometric meta-analyses have helped organizational researchers integrate large quantities of research in a systematic way, account for measurement error and other statistical artifacts (e.g. range restriction; Ones and Viswesvaran, 2003), and to produce cumulative science that has advanced both knowledge and practice (Michel et al., 2011; Ones et al., 2017; Stone and Rosopa, 2017).

Furthermore, as the study of career management develops, the psychological theories postulated to explain the process mechanisms underlying the outcomes (e.g. career satisfaction, job search behaviors, and workplace behaviors) have also grown in complexity. For example, recent decades have seen an increase in studies exploring the role of personality traits in explaining career satisfaction. The number of antecedents explored in these studies has increased, as have the moderators and mediators that define the boundary conditions for the validity of the psychological process mechanisms. It is increasingly difficult to test such complex models in one primary study, necessitating the researchers to abstract a simpler model for empirical testing. With meta-analytic cumulation, one can build a meta-analyzed correlation matrix among variables not all measured in the same primary study. Such meta-analyzed correlation matrices can be employed to test more comprehensive theories of career management. For example, several theories of workplace behavior have been based on meta-analytic structural equation modeling (MASEM) (Cheung and Chan, 2009; Jak, 2015; Michel et al., 2011; Viswesvaran and Ones, 1995). Indeed, quantitatively grounded theories can be built by combining psychometric meta-analyses of multiple bivariate relationships and using a matrix of meta-analyzed correlations to test alternate structural models of workplace behavior (cf. Viswesvaran and Ones, 1995). In the fields of industrial, work, and organizational psychology and career development, there are by now many example applications of MASEM, including tests of the theories of job performance (Viswesvaran et al., 2005), the role of general mental ability in predicting work performance (Schmidt et al., 1986), the theories of voluntary turnover (Hom et al., 2012), influences of emotional intelligence in workplace behavior (Joseph and Newman, 2010; Van Rooy and Viswesvaran, 2004), etc. These methods have been applied to test the process mechanisms by which workplace interventions work.

When meta-analytic methods are employed to integrate findings across several primary studies, we can also investigate how the bivariate relations are affected by study-level characteristics. The primary idea underlying this use of meta-analysis is that when results across studies are cumulated, we better understand the relations among constructs (at the population level) than would be possible through individual primary research (Bobko and Stone-Romero, 1998). Meta-analyses have helped HR practitioners and organizational consultants to present empirically based scientific interventions to their clients, and meta-analyses have become seminal papers in our field due to their ability to speak to theory development and identify profitable paths for future research. We also note the vital role meta-analyses play in guiding evidence-based practice.

Motivation for this special issue

Against this backdrop, we identified a number of areas where either an open discussion or some useful examples could be useful in furthering applications of meta-analyses in career- and work-related research. This included some issues that researchers and the field in general may need to be more cognizant of. Meta-analysis is a powerful tool to advance our understanding of career management. One of our aims in this special issue is to draw attention to issues associated with the use of meta-analysis and to illustrate how they can be successfully handled/addressed. Let us focus on two key issues.

First, the generalizability of the database used in any meta-analysis is a primary concern. With new studies being continuously published, we need to investigate how generalizable the conclusions in a meta-analysis are. Related to this issue, new developments in data storage are becoming the norm. The medical professions have long pioneered the concept of a repository of effect sizes (e.g. Cochrane collaboration) and the recent push in replicability of studies in social sciences have also spawned databases of effect sizes (e.g. MetaBUS). The globalization of research presents meta-analysts with large data sets drawn from different countries, consulting companies, etc.; successful integration of these data sets is a real challenge (cf. Ones et al., 2017 for an elaborate discussion). Indeed, in our view, most meta-analyses are rarely comprehensive in that a vast volume of invisible literature is never in the meta-analytic database. In toal, five of the six studies in this special issue grapple with this problem and present some unique and novel ways to ensure a comprehensive coverage. Overall, the hope is to either illustrate the robustness of the findings reported in the published literature or to highlight their dependency on the meta-analytic databases/analyses undertaken (Spoiler alert: in fact, it is quite heartening to note that even with the novel data searches and inclusion, at a broad-brush level, the general findings are found to be robust to what has been reported in the published literature! But a handful of nuanced findings indicate a need to continue incorporating multiple data sources and searches to meta-analyses to refine theories and applications).

A second focal area is to raise awareness about some methodological issues that authors need to be aware of. Specifically, we caution authors to more accurately consider justified moderator influences and/or to better tease apart the influences of correlated moderators (e.g. through hierarchical moderator analyses). Every published meta-analysis has considered moderator influences, and quantifying the effect of such moderators is an important and central issue in any meta-analysis. Sadly, most meta-analyses have employed misleading or erroneous approaches. These are issues tackled head-on in two of the articles in the special issue.

Papers included in this special issue

In total, six papers that speak to current issues in the use of meta-analyses for assessing empirical relations and theory building have been included in this special issue. These papers have direct implications for I-O psychology and organizational behavior literatures in general, but also for career management research and practice.

Schmidt presents the case why meta-regression results in erroneous conclusions when applied to test for the relative importance of moderators. In a typical meta-analysis, meta-analysts employ the effect sizes as the dependent variable and the set of hypothesized moderators as the independent variables in regression analyses. The β weights are taken as an index of the relative weight of the moderators (and oftentimes the statistical significance of the β weights are taken as an indicator of the presence and significance of the moderators). Schmidt points out that in addition to the problem of small sample size in the regression (here, the sample size is the number of primary studies, and more problematically, the number of primary studies that included the moderators assessed in the regression – which is usually much smaller than the number of primary studies in the meta-analyses), we have issues such as ex post selection of predictors. This is a universal problem in meta-analyses, and we hope the article by Schmidt will spur researchers to avoid such pitfalls in moderator searches. We also point the interested reader to the general problem of interpreting regression weights at all: regression weights are fungible and therefore should not be relied upon (Waller, 2008).

Seltzer, Ones and Tater illustrate how bi-factor latent modeling can be employed to tease out the unique effects of an antecedent when it shares covariance with several related antecedents that can be conceptualized as reflecting a higher-order factor. It has been postulated that multi-tasking behaviors are affected by several personal and situational factors (Sanderson et al., 2013; Mesmer-Magnus et al., 2014). Many of the situational factors can be modeled as being caused by an underlying latent factor. The same holds with personal factors. Personality traits like sociability, achievement motivation, dominance, etc. can influence multi-tasking with different traits influencing multi-tasking through different mechanisms. The unique influence of these traits, independent of any underlying general construct, can be assessed with the bi-factor latent modeling. Seltzer et al. show that the process mechanisms underlying the personality-satisfaction relationship differ across facets using bi-factor latent modeling, applied to meta-analytic data. Thus, although applied prediction is better served by the general underlying trait (Ones and Viswesvaran, 1996), i.e. better understanding results by examining the “unique” influence of the facets. Seltzer et al. state “Bi-factor models isolate the variance associated with the general factor in a domain (which is shared across facet measures) from the variance that is unique to each facet measure. In doing so, bi-factor models allow the unique contributions of facets to criterion prediction to be examined separately from the contribution of the general factor (cf. McAbee et al., 2014; Salgado et al., 2015).”

Meta-analytic conclusions depend on the completeness of the primary literature that has been included within their databases. In many areas of applied research, new empirical studies that report a bivariate relationship, even after a meta-analytic cumulation of that bivariate relationship, are often published. For example, several studies have reported the correlation between the personality trait of conscientiousness and the job performance of employees in the last two decades even though several meta-analyses have robustly established the conscientiousness-job performance link (Barrick and Mount, 1991; Salgado, 1997). The publication of bivariate relationships after a meta-analyses may continue because the focus of the paper is on different topics but the bivariate relationship is an incidental part of the analyses, or the authors found a unique boundary condition where the bivariate relationship summarized in the meta-analysis does not hold. This latter reason includes the boundary condition of temporal change in the relations being examined. In either case, it is interesting and important to know whether the results of seminal bivariate relationships are influenced by the inclusion of new primary studies. Our hypothesis is as follows:

H1.

For most meta-analyses of bivariate relationships based on large data sets (Ones et al., 1993), such reversals are unlikely – the conclusions of the meta-analyses are likely to be robust.

For example, in this special issue, Mesmer-Magnus, Niler, Plummer, Larson and DeChurch report an updated meta-analysis of the role of team cognition in team process and performance. In 2010, DeChurch and Mesmer-Magnus published a comprehensive meta-analysis of team cognition’s role in team functioning. Since this publication, research on team cognition has exploded in the primary literature warranting an updated meta-analysis wherein the meta-analytic database would include the current extant literature in order to determine the extent to which our understanding of team cognition’s role in team functioning still holds, given the new research and the vastly changing scope of contemporary work teams. Interestingly, although the majority of findings hold, the authors note one key relationship that has changed along with the changing landscape of teams: they discuss the implications of this development in their manuscript.

A key challenge faced by meta-analysts relates to variations in the internal, external, construct, and statistical conclusion validity of the studies included in the meta-analytic database (Stone and Rosopa, 2017). The competency and consistency of the meta-analysts’ coding of the primary studies included in the database affects the soundness of the conclusions that can be drawn based on the results. Meta-analyses allow researchers to apply new theoretical frameworks to the integration of the primary studies. Kostal and Wiernik, in their paper,integrate different conceptualizations of protean careers and illustrate how definitional issues can be handled theoretically and tested empirically. The authors note how this concept has encompassed not only economic conditions but also employee behaviors. Broad thinking and careful integration, including attention to measurement details of articles included in a meta-analysis, is essential for theoretically and practically useful meta-analyses.

Meta-analysis is usually applied to areas of research that have been studied for decades – but this need not necessarily be the case. Several new topics of interest have emerged in career management and in applied psychology in general. The profession can usually draw from several related areas of enquiry into a question. Schmidt (2008) points out how the failure to consider data from behavior genetics has resulted in the study of individual differences focusing on a limited empirical evidential base. This problem becomes magnified in meta-analytic results when conclusions are drawn based upon an incomplete picture of what is available in the extant literature. Relevant research is being conducted in a wide variety of disciplines and published in a wide variety of outlets. When meta-analysts fail to collect studies from these various disciplines, they risk drawing conclusions that do not actually represent the broader population. For example, research related to personnel selection may be published in the human resources, organizational behavior, and industrial/organizational psychology literatures; however, relevant research is also being conducted in military, healthcare, and education settings as well as in languages other than English. Failing to canvas the complete literature threatens the validity and generalizability of meta-analytic results (Ones et al., 2017). Further complicating matters is the fact that some of this primary research is being conducted within organizations by practitioners who do not publish their findings due to the lack of time and incentives or because the information is proprietary. Mercado, Giordano and Dilchert illustrate how meta-analysts can cumulate data from multiple sources and fields to address current topics – in their example, they assess the correlates of cyber-loafing, a behavioral domain with potentially career consequences. We commend Mercado et al. for opening an avenue for career management research.

On a related note, databases are being created to function as a repository of effect sizes reported in the main journals focused on applied psychology. The MetaBUS (Bosco et al., 2017) has garnered much attention in human resource management, and the Lee et al.’s paper illustrates the strengths as well as potential issues in utilizing this valuable database resource in career management scholarship. Specifically, the authors provide testable hypotheses when results from MetaBUS analysis differ from earlier published meta-analyses.

Finally, all the papers in this issue stress the need for meta-analysts to ensure that they are appropriately correcting for statistical artifacts. Although artifacts other than sampling error do not tend to substantially inflate observed variability, they do result in significant underestimation of effect sizes (Ones et al., 2017). Since meta-analytic methodologies are a set of constantly evolving research integration tools (Schmidt, 2008), it is wise to replicate foundational meta-analyses by updating their databases of primary studies and evaluating the appropriateness of the techniques used to correct for artifacts and explore moderators. In particular, meta-analytic conclusions can be affected when the meta-analyst fails to include relevant primary studies and/or uses inappropriate corrections or analyses to examine moderator variables (Schmidt and Hunter, 2014).

Conclusion

In sum, we hope the reader finds the assembled collection of articles to be interesting and informative. We hoped to elucidate the practical and theoretical sense of meta-analytic applications. We want to take this opportunity to thank Jim Jawahar for giving us the opportunity to edit this special issue for Career Development International. His support and insight has been instrumental in getting the special issue compiled. We are also grateful to the authors who submitted manuscripts for consideration for this special issue and their careful attention to suggested revisions. Finally, we are very grateful for the time and attention provided by the peer reviewers who spent countless hours evaluating submissions and providing insightful recommendations for revisions. We have learned much by reading these submissions and hope the reader will find this special issue enlightening and useful in future research and applications.