Guest editorialNovel and overlooked techniques for public management research questions

Abstract

Purpose

This article provides a snapshot of several innovative and underused methodological approaches employed by scholars from across public management.

Design/methodology/approach

A comparative review of methodological approaches to public management research is used.

Findings

The authors find evidence of scholars applying numerous novel methodological approaches to study social science phenomena, including agent-based modeling, nonparametic approaches, social network analysis, Granger equations and techniques for correcting selection bias.

Research limitations/implications

The review does not cover all of the innovative methodological approaches used in social science. However, the methodological techniques showcased offer promise for advancing public management research, whether used as primary applications or as one of multiple methods (triangulation) to test the validity of research findings and/or to more comprehensively understand the phenomena being studied.

Originality/value

The techniques reviewed include example applications to aid researchers who may wish to apply them in their own research. The novel tools and methods already in use by public management researchers contradict the perception that public management methods have lagged behind other social science disciplines.

Citation

Malatesta, D. and Smith, C. (2021), "Guest editorialNovel and overlooked techniques for public management research questions", International Journal of Public Sector Management, Vol. 34 No. 6, pp. 605-613. https://doi.org/10.1108/IJPSM-08-2021-348

1. Introduction to the special issue

This special issue began with a call for manuscripts focusing on innovations in public management research methods. Whether warranted or not, there is a perception that public management methods have lagged behind other social science disciplines (Gill and Meier, 2000). Not surprisingly, there has been considerable discussion about how to develop more rigorous and innovative approaches to public management research (for example, Grimmelikhuijsen et al., 2017). The premise for the issue is aligned with one of IJPSM's principal aims: to provide a snapshot of some of the unique methodological approaches employed within the field. Our goal from the outset has been to highlight the novel methodological approaches with the hope that it will inspire others to consider the numerous tools and methods available to them. While our call for papers did not distinguish between qualitative and quantitative methods, the abstract proposals were almost entirely quantitative in nature. As such, the included manuscripts focus primarily on quantitative approaches to public management research. Despite the lack of qualitative methods, the special issue achieves the goal of showcasing unique methodological approaches. In this regard the manuscripts did not disappoint.

Novelty in research methods can take on many forms ranging from introducing a new or not commonly used estimation technique to rethinking our assumptions about commonly used techniques. For example, Jones et al. make a strong case for less reliance on standard parametric approaches for dichotomous outcomes, such as logit and probit, and more consideration of nonparametric and semiparametric approaches. He argues convincingly that utilizing models with fewer a priori restrictions opens up possibilities for a wider range of theoretical explanations. The manuscript by Christian Buerger and his colleagues offers an alternative to the standard difference in difference (DiD) method. Specifically, the authors demonstrate the use of Granger equations for establishing causality. The application is based on a salient topic, “cutback management,” which concerns the processes and strategies used by policymakers when their organization faces a situation of resource scarcity. Unlike the previous manuscripts that recast traditional methods, Choi and Park offer an alternative modeling method. The manuscript highlights the efficacy of agent-based modeling (ABM) for public administration research questions. Choi and Park define ABM as “a research method used to model how system-level characteristics emerge from complex local interactions among agents.” Based on our search of articles in SCOPUS and Web of Science, over the last ten years fewer than eight papers employ agent-based modeling in public management research.

The Nam-Speers and da Fonesca manuscripts also illuminate novel methodological approaches. Nam-Speers presents several common problems in social science such as obtaining valid and reliable measures when survey questions call for disagree-agree responses. Making important decisions based on attitude can be particularly problematic when a respondents' attitudes are on the cusp between response categories. Although the suggestion is somewhat buried in the paper, Nam-Speers calls attention to advances in item response theory (IRT) that have led to the development and use of several parametric unfolding models that can be used for graded responses (Roberts and Laughlin, 1996). The da Fonseca manuscript provides an overview and an exploratory social network analysis (SNA). Although SNA is not considered a new method in public administration research the author's main contribution is how to apply the method in a new and meaningful way. By doing so, he is able to help translate a complex method in such a way as to provide meaningful and practical implications.

In sum, the manuscripts help demonstrate that novelty is more than providing new and better methodological techniques; it is also about reexamining our current research paradigms in an effort to better serve the field. In the next section we provide a brief overview of the manuscripts and highlight the unique contributions of each.

2. Overview of the manuscripts

In Theory Building in Agent Based Modelling in Public Administration Research: Vindications and Limitations, Choi and Park provide a useful introduction to the efficacy (and pitfalls) of agent-based modeling (ABM). Choi and Park begin by providing an overview of ABM, paying close attention to its epistemological grounding. In a previous paper, Choi and Robertson (2014) define ABM as “a research method used to model how system-level characteristics emerge from complex local interactions among agents” (501). Based on this definition, ABM can be viewed, epistemologically, as both generative/emergent and computational. It is emergent and generative in that a simple set of rules governing the actions of agents can result in an emergent set of social phenomena. Indeed, Choi and Park point out that some scholars go so far as to view the subsequent model as equivalent to theory. ABM's computational qualities, through simulation, allow for differing results depending on how the initial parameters are defined, and such differences can lead to surprising conclusions, similar to other formal approaches to modeling.

Unlike the other manuscripts in the issue that attempt to improve measurement, clarify assumptions or recast existing methods, Choi and Park recommend ABM as an alternative modeling technique, which is well-suited to address increasingly complex governance schemes due to so-called wicked policy problems. Moreover, Choi and Park argue convincingly that ABM provides researchers with considerable leverage, in part, due to its ability to incorporate the behavioral aspects of administration. She contends that recent focus on behavioral public administration has led to greater integration of concepts and methods from psychology in traditional public administration settings. ABM provides a platform where “psychologically derived methodological devices can stand in parallel with other known tools to arrive at realistic approximation of dynamic real world” and thus “elevate its role in responding to the rise of murky and unpredictable issues of today.”

At its core, ABM rests on three primary building blocks: agents, rules and the environment. In simplest terms, agents interact under a set of constraints imposed by rules and/or the macro-level environment. The interactions lead to a resulting structure, which, in theory, can be explained by the internal mechanism; that is why the model produces this structure. Choi and Park point out that like many methodological approaches, there is a tradeoff between the transparency of the model and the tractability of the internal mechanism – the more complex the internal mechanism, the more difficult it is to understand the implications of the model.

Choi and Park assert that ABM's utility to public administration scholars rests on its ability to approximate dynamic contexts. Choi and Robertson's (2014, 2019) research on decision-making and Scott et al.'s (2019) research on collaborative governance are just a few examples of how ABM might be utilized within public administration and management. Collaborative governance, a research topic often explored in IJPSM, seems like a particularly fruitful research domain. Formal collaborative endeavors are characterized by agents with varying levels of power, differing level of homophily and goal asymmetry. All of this takes place within an institutional environment and under a set of rules. Modelling behavior subsequent to changes in such formal constraints could help researchers improve institutional design.

While Choi and Park are sanguine about the utility of ABM in public administration and management, she is quick to point out its limitations. Most notably, the results from ABM simulations should not be viewed as a substitute for empirical results. Instead, researchers may gain more traction by treating ABM and empirical findings as complementary since agent behavior can be modeled based on empirical results, and ABM modeling can be used to derive testable propositions and hypotheses for subsequent empirical analysis. In this regard, Choi and Park’s contribution is also theoretical. Still, some topics are simply better analyzed through traditional empirical techniques. For example, testing whether individuals are better motivated by money, gift or compliment (see Gneezy and List, 2013), seems to be better served by simple empirical tests. In sum, Choi and Park provide a gentle, yet informative, introduction to the utility of ABM in public administration and management. Its methodological appeal stems from its ability to control and simplify agent interactions within complex domains. The potential for unexpected outcomes of such interactions provides leverage for scholars examining topics that are beset by complexity, such as collaborative governance, network governance and decision-making.

A manuscript by Jones et al., Going Beyond Parametric Regression in Public Management Research, aims to compare parametric, nonparametric and semi-parametric estimators and to illustrate differences among the three approaches when modeling dichotomous variables. The standard logit, probit and linear probability models rely on the data-generating process to conform to parametric assumptions. While most researchers acknowledge that ordinary least squares regression produces biased estimates when modeling binary dependent variables (i.e. linear probability model), they are less likely to question the parametric assumptions underlying logit and probit models. Indeed, researchers commonly choose the type of modeling based on how much we know about the form of the relationship between dependent variable and explanatory variables, and the randomness of the error distribution. Yet it is probably safe to say that the fall back and default choice is the standard parametric regression. Jones et al.’s primary argument is that while these parametric models may be computationally quicker and easier to estimate than nonparametric models, it comes at a cost since the data-generating process and parametric assumptions are often not aligned. In contrast, nonparametric and semi-parametric models offer flexibility with regard to such conformity. Specifically, parametric models require fewer assumptions regarding the distribution of underlying data or the error spread. However nonparametric models are computationally slower compared to parametric models.

Jones et al. also make a strong pitch for the consideration of nonparametric and semi-parametric approaches, which combine features of parametric and nonparametric models to form a hybrid approach. Over the last decade, increasing attention has been devoted to these regression models as new techniques for estimation because they relax the assumptions about the underlying data, functional form and random error distribution. In support of their claims, Jones et al. use a three-step strategy to compare estimators. First, they estimate a linear probability model, a probit model and a logit model and then use those results to estimate a maximum score (semiparametric) model, based on the least absolute deviation estimator, which is the less restrictive model of dependent variables. Next, they use results from the maximum score models to generate a single index, the sum of explanatory variables times coefficients, to use in a nonparametric regression. Jones et al. rely on a previous study with a dataset of 2,179 bond referenda for school districts in Texas over a time period of seven years, from 1998 to 2015. The previous study focused analysis on whether voters anchor their decisions to support bond referendum vote on school district performance ratings and examined if changes in ratings (upgrade or downgrade) influenced the likelihood of the bond getting passed. Results from the study suggested voters anchored their vote in to pass performance though results varied across different performance measurement systems. Jones et al.’s focus is on the accuracy of the models and consideration of whether complex semi-parametric procedures are capable of better predictions than standard models. The author shows how to leverage the nonparametric technique with the bond data using a maximum score estimation, which was introduced for discrete choice models by Charles Manski in 1975 (Manski and McFadden, 1981).

Finally, Jones et al. do acknowledge that nonparametric and semi-parametric models are not always optimum. To be sure, nonparametric regression requires larger sample sizes than parametric regression because the data supply both the model structure and the model estimates, as well as require less common statistical programs such as LIMDEP (LIMited DEPendent variable models). However, Jones et al. make a convincing argument for the consideration of nonparametric and semi-parametric approaches by public management researchers.

In Exploring Government Networks through Inter-Organizational Relationships: Research Strategies based on Social Network Analysis, da Fonseca provides an overview and an exploratory social network analysis (SNA). The manuscript begins by making the connection between networks and managerial strategies. Implicit is the assumption that managing in networked environments is distinctive. Building on the work of Klijn et al. (1995), da Fonseca characterizes managerial strategies as “games” within networked structures. Both the network's game strategy and structuring are influenced by the different aspects of the network: actors, resources, rules and perceptions. The key insight is that managers respond to – and shape – strategies and structure in response to variation in resources and perceptions. What emerge are distinctive modes of governance. Shared governance occurs when the actors make collectively responsible decisions within the network. Cooperation emerges when there is little disparity in power and resources across actors. Contrarily, led-network governance emerges when one actor has considerable influence reducing other actors to a state of dependence. Finally, managed governance occurs when there is a purposive managerial strategy implemented within the network, specifically in the form of a network administrative organization (NAO).

This manuscript differs from the others in that SNA is not necessarily a new method in public administration research. Indeed, the idea that scholars should “treat networks seriously” (see O'Toole, 1997) is well entrenched in the literature, as well as in the public administration psyche. Thus, the manuscript's emphasis is less about introducing a new method and more about how to apply the method in an advantageous way. Principally, da Fonseca argues that analyzing at the network level is more beneficial than focusing on organizations within networks, which thereby renders SNA the most appropriate analytical tool. By providing a set of practical considerations for engaging in SNA, he hopes to stimulate more whole-network analyses.

The manuscript's most important contribution is not its description of governance regimes, but rather in providing a roadmap for an exploratory SNA. Specifically, da Fonseca conducts an analysis of training programs within a network of Brazilian federal agencies that are responsible for the implementation of the National Personnel Development Policy (NPDP). The analysis provides the reader with practical examples of how to conceptualize the boundaries of the network, better understand the network variables (i.e. actors, resources, rules and perceptions), assess and visualize the characteristics of the network (e.g. network centrality) and identify subnetworks or groups within the network.

Finally, da Fonseca discusses the empirical implications of the SNA. Most notably, he finds that the network follows a led type governance structure. Specifically, the analysis identifies two primary organizations responsible for training within the network. Linking the analysis back to the network governance regimes, he posits that changing the rules within the network may provide a pathway for better distribution of training throughout the network. In this sense, the manuscript highlights how methodological tools can bridge theory and practice.

In Simulating Self Selection in Public Management Research: Implications from Caseworker Discretion in the Child Welfare System, Nam-Speers confronts an important measurement problem in public management research – self-selection bias. She begins by focusing on problems associated with categorical data of the Likert-scale type. To describe the challenge in concrete terms, Nam-Speers asks readers to consider the “A” through “F” grading scale familiar to instructors and the case of a student on the borderline of two letter grades, a “D” and an “F”. As she explains, the decision to round the grade up to a “D” overrides and likely misrepresents the underlying information. This type of bias disproportionately affects the students on the border of two grades, near the cut point, rather than the large student population.

Nam-Speers then turns to a bias typical of surveys and some types of survey questions. Consider a survey question asking about a student's level of agreement on the statement “The course fulfilled my expectations.” The response options are situated on the familiar continuum, including options strongly agree, agree, neutral, disagree and strongly disagree. Questions and responses of this nature tend to illicit psychological traits or attitudes. Similar to the challenge with the letter grade system, the problem in measuring this type of survey response arises more often at the cut-points than anywhere else, which is to say those students with opinions between two response options, for example, students may be mostly satisfied with a course, yet are hesitant to report that the course was entirely fulfilling. These issues are pervasive in social science research and their implications raise questions about whether we are actually measuring what we think we are measuring. Yet it is only when we consider possible ramifications beyond a student's final course grade or an opinion on an end-of semester course evaluation do we recognize the criticality of mismeasurement and bias. Nam-Speers brings this to the forefront in the example of a caseworker's application of discretion in decisions about child placement in foster care. The decision to place a child in foster care undoubtedly affects their life prospects. Moreover, the research indicates these discretionary decisions tend to disproportionately affect children of minority groups.

Although there are different methods for determining whether a child is placed in a foster home, a common method relies on a “care continuum,” where caseworkers use a score-based system to “judge” the risk of abuse or neglect if a child remains in his/her home. A child's latent score is manifest as a continuous variable (just as with the student's underlying course grade before it is converted to a letter grade). For the child risk assessment, when the risk score is higher than a determined marker, a cut point, the child is placed in foster care, otherwise he/she remains in the home. Child placement, therefore, is determined once the measure is reduced to a dichotomous zero/one measure. A rounding up of the score, a common practice for borderline assessments can change more than a course grade; it can change the course of a child's life.

Having laid out the criticality of the initial measurements and its ramifications, Nam-Speers moves on to assess outcomes, that is, how well children fare after placement compared to those children who remain in–home. The challenge here is assessing bias associated with those who have been selected for foster care. The size of the bias depends on how close the initial score was to the cut point, thus disproportionately affecting those children considered on the borderline. The question (and analytical strategy) becomes how to determine and then account for the selection bias, which is the difference in pretreatment outcomes for those treated (transferred) and those untreated (those who remain in the home).

A Monte Carlo simulation is one approach for understanding a statistic's sampling distribution and evaluating its behavior in random samples. Starting with a base model of most likely values, it uses repeated random sampling. In Nam-Speers' study, the process was repeated until the transfer percentage converges on the target value by generating sets of random numbers for each specified distribution and then different sets of parameter estimates as possible output values”. A second approach relies on a DiD estimation. The DiD technique attempts to mimic an experimental research design using observational study data, by studying the differential effect of a treatment on a “treatment group” versus a “control group” in a natural experiment. DiD calculates the effect of a treatment by comparing the average change over time in the outcome variable for the treatment group to the average change over time for the control group. Both approaches are common to social sciences and are intended to mitigate the effects of extraneous factors and selection bias.

Nam-Speers successfully convinces that selection bias is embedded in the models that rely on measures such as those reduced from continuous scores to dichotomous choices, and that the cut points are particularly problematic in this regard. Both Monte-Carlo and DiD approaches have their advantages and disadvantages, and her aim is not to declare one superior to the other. As for the specific context of child welfare decisions, the analytics show that differential impacts depend not only on where or if a child is placed, but critically on the design of the decision for placement. Despite the shortcomings of both approaches that are noted by the author herself, Nam-Speers wisely calls for public management researchers to rely on multiple techniques to weed out biases before drawing conclusions. Lastly, Nam-Speers brings attention to a simulation study and unfolding model introduced by Roberts and Laughlin (1996), a latent trait model popularly used in psychology but not yet widely used in public management research, for explaining nonrandom factors embedded in survey item analysis that are common scores reduced to more crude measures.

Buerger et al. contribute Extending Differences-In-Differences Frameworks to Granger Equations: Evidence from Cutback Management during Three Recessions. This manuscript prompted us to question the often-expressed claim that “past is prologue.” This manuscript presents a particular type of causality, Granger causality, which is useful for unpacking the relationships between past and present (Hamilton, 1994, pp 306–308). Granger causality has been described as focusing on “precedence” or, as Granger and Newbold (1977, p. 225) himself described, “temporally related”. Rather than testing whether X causes Y, the Granger causality tests whether X forecasts Y. According to Granger logic, a time series X is said to Granger-cause Y if it can be shown, usually through a series of t-tests and F-tests on lagged values of X (and with lagged values of Y also included), that those X values provide statistically significant information about future values of Y. Thus, the Granger causality test provides a hypothesis for determining whether one time series offers useful information in forecasting another time series.

Researchers often employ DiD methods to estimate treatment effects by comparing the change or difference in observed outcomes between treatment and control groups, across pretreatment and posttreatment periods. Buerger et al. submit the advantages of Granger equations to evaluate changes between groups. The approach is a modification to the classical DiD framework and is applicable to a variety of topics in public administration if the researcher has panel data and both a treatment and control group. According to the authors, Granger equations have several features that make them preferable to other DiD strategies. Most notably, Granger equations require only two pre-period observations. The specification offers a statistical test for parallel trends, that is, determining whether the treatment and control group had common trends prior to the start of the treatment. Granger equations are easily implemented with commonly used software such Stata or R.

The authors extended the DiD modeling approach by implementing Granger equations to measure the impact of recession on state budgets. Specifically, Berger et al. illustrate how Granger equations can improve causal claims with financial data from American states during the periods 1990, 2001 and 2007. The authors find that states deepen expenditure cuts when economic declines lengthen, while states tend to make larger cuts to specific expenditure categories instead of spreading cuts equally over all service areas.

In terms of methodological contributions, the authors explain the use of Granger equations for testing the parallel trend assumptions and measure changes in treatment effects over time. They demonstrate the use of linear combinations to evaluate differences between pre and post coefficients in the Granger equations to adjust results for opposite trends in the pre period. They provide a detailed discussion on how to calculate percentage counterfactuals to compare outcomes with different baseline magnitudes, which is particularly important for research that draws distinctions between nominal and relative effects of interventions or events. Finally, they discuss and implement several techniques to mitigate omitted variable bias that can be incorporated into other research designed to estimate causal effects of policy or organizational change.

3. Additional emergent themes

In addition to the novel methods and applications presented in the manuscripts, a couple of additional themes emerge when they are examined comparatively. First, there is need for reliable and valid measurement, which is foundational to effective social science research. And, innovations in measurement – and our assumptions undergirding it – are at the forefront of contemporary public management research. While two of the manuscripts address the issue explicitly (Jones et al.; Nam-Speers), the others highlight its importance more indirectly. Arguably, measurement is the sine qua non of quantitative methodology. However, no amount of sophisticated statistical analysis can overcome a poor measurement scheme. Accordingly, we encourage researchers to attend to the basics, such as focusing on valid and reliable measures.

Some of the foundational ideas of the behavioral theory of the firm (BTF; Cyert and March, 1963) peek through several pages of the manuscripts in this special issue. For example, main ideas behind Granger causality in the Buerger manuscript can be connected to Cyert and March's claims regarding how organizations use (and misuse) past information. More specifically, Cyert and March conceived of expectations as a result of drawing inferences from available information; they also claimed that organizational decisions depend on information estimates and expectations that often differ appreciably from reality. The Granger causality tests explored by Christian Buerger can be put to use to explore some of these claims. According to Cyert and March, a sound theory for organizational expectations should account for factors that may affect either the process of drawing inferences or the process by which information is made available to the organization; and, a theory of organizational choice should characterize the process by which the alternatives available to the organization are ordered and selected. Indeed, scholars of public management often refer to the BTF in broad terms but seldom call on these essential details for theory development (Rainey et al., 2021). In our opinion, renewed consideration of BTF's details is not only warranted but necessary to advance public management theory.

Cyert and March were not only concerned with the internal workings of the organization but also with the relationships between the organization and its external environment. As a method, SNA focuses on patterns of relations among people and other organizations. Da Fonesca's manuscript provides a useful example of an SNA application. Although SNA is primarily viewed as a method of analysis, the possibilities for advancing theory with SNA are promising. No doubt the graphics in da Fonesca illustrate how connections and interrelationships among actors, resources and institutions that may not have been initially evident to the researcher become evident once they are viewable. Once revealed, researchers can use the visual connections provided by SNA to better understand conditions that may be necessary (or not) for fruitful collaboration.

Several manuscripts also align with the recent resurgence of interest in behavior research in public administration and management. From Waldo and Simon, to the organizational insights of the Carnegie School (Cyert and March, 1963), public administration scholars have not shieded away from behavioral research. However, in recent years there has been newfound energy in behavioral approaches, highlighted by a coalescing of scholarship within the subfield of behavioral public administration (BPA) (see Bhanot and Linos, 2020 for overview). A recent special issue in the Journal of Public Administration Research and Theory (Tummers et al., 2016) captures both breadth of scholarship in BPA as well as the energy driving these research efforts. Of the several methodological tools explained and employed in this special issue, ABM (see Choi and Park) stands out as very promising for advancing our understanding of actor behaviors. ABM conceives of social agents as evolving systems of autonomous interacting agents. From this perspective, the researcher can explore how macro phenomena might emerge from micro-level behavior among a heterogeneous set of interacting agents (Holland, 1992).

Lastly, complexity is an additional emergent theme. Two manuscripts, Choi and Park and da Fonseca, attempt to capture the realities of complex managerial and interorganizational systems, yet they approach complexity from different vantage points. In ABM, social systems are viewed from the complex adaptive system perspective. Thus, Choi and Park focus on the micro and behavioral dimensions of complexity via ABM, while da Fonseca emphasizes the need to capture the complex interorganizational relationships in contemporary governance through SNA. Both methodologies can be used to develop hypotheses and to test agent attributes, their behaviors and behavior rules, interactions and interdependencies and their effects on the macro-level-stylized facts of the system.

In conclusion, we thank all of the researchers who contributed to this special issue of IJPSM for sharing their work. We view each manuscript as an important step to negating the perception that public management methods have lagged behind other social science disciplines (Gill and Meier, 2000).