Editorial: Why should we care about preparing referee reports?

Citation

Albergaria, M., Hourneaux Junior, F. and Dionizio Leite, P.F. (2021), "Editorial: Why should we care about preparing referee reports?", RAUSP Management Journal, Vol. 56 No. 4, pp. 386-389. https://doi.org/10.1108/RAUSP-10-2021-0195

It is undeniable that the whole academic publication universe depends on the fundamental, but not necessarily enough recognized and rewarded, reviewers’ role. The fact is that without the reviewers, even if we consider the new open science practices, there is no way to guarantee the quality of the published research in journals worldwide (Bedeian, 2003; Gerhardt, Dal Pai, Gouveia & Azzolin, 2014).

Although authors have praised the role of anonymous reviewers in academic life (Hamermesh, 1992, 1994; McAfee, 2010; Taylor, 2009), there seem to be very few defined and consensual reasons for scholars to prepare referee reports promptly (Tosi, 2009). In fact, according to some studies, delays and missed deadlines are often a common feature of academic publishing today (Ellison, 2002; Card & Dellavigna, 2013; Hamermesh, 2013; Gerhardt et al., 2014).

If this is the general context and understanding, why should scholars care about writing referee reports? Starting from this doubt, this editorial aims at discussing the motivations behind the reviewers’ commitment to performing their tasks.

The reviewer’s dilemma

How does the process of preparing a report work? In general, preparing a referee report is an anonymous [1] task, for referees are not known for the service they perform (Gerhardt et al., 2014). More than that, the preparation of such reports is unpaid and can be time-consuming for those involved in such a task (Chetty, Saez, and Sándor, 2014; Gerhardt et al., 2014). Thus, for example, when scholars must decide how to divide their time into various academic tasks such as class preparation, research activities or committee participation, among others, it makes sense to imagine that the preparation of referee reports may not be one of their priorities (Berk, Harvey & Hirshleifer, 2017; Hamermesh, 1992, 1994; Tosi, 2009).

Moreover, if the reviewers’ value perception of the evaluation process is not positive, there could be some distortions along the way (Bedeian, 2003; Tosi, 2009). The quote below reflects one example:

At a recent reviewer workshop I attended, one reviewer lamented that although providing such detail was not terribly difficult for one who knows the research domain, “Why should I do the author’s work for him?” A fair question, and one that a panelist answered simply, “Why not?” After all, a reviewer’s expertise is exactly the value she brings to the process (Schminke, 2002, p. 487).

In other words, in situations like this, an academic can ask the following question: “Will my actions have any significant impact on the remainder of the academic community?” If the answer is negative, this person is likely to have little incentive to cooperate with the collective undertaking as his/her actions are perceived as socially insignificant. One problem derived from such thinking is that if many people proceed in this way, it may be difficult for the academic community to achieve collective goals, such as advancing knowledge in the scientific community. After all, the prosperity of the scientific community is based on referee reports (Treviño, 2008; Gerhardt et al., 2014).

Then, what kind of incentives could matter?

Given this context, some questions arise: What would be the best way to encourage the preparation of referee reports for academic journals? Would it be worth paying reviewers for such a task? Moreover, what would be the proper monetary values to encourage the preparation of reviews with enhanced quality? Conversely, would there be alternative ways – besides monetary payments – to encourage this type of prosocial behaviour?

There are examples of situations in which the introduction of monetary incentives ended up generating unintended effects. For example, a day-care centre in Israel tried to establish a fine for parents of children who remained there after official departure hours. The initial intention of the day-care administration was to encourage parents to pick their children up on time. Ironically, what ended up happening after the introduction of the fine was an actual increase in the average time of delays (even after the end of the fine). That is, instead of reducing parents’ delays, the fine ended up raising them as some parents saw it as a price for lateness (Gneezy & Rustichini, 2000).

Despite these limitations of monetary incentives, why not run an experiment to test the best way to encourage the preparation of referee reports on time? That was precisely the proposal by Chetty et al. (2014), in which the authors ran an experiment with peer reviewers from an academic journal in the field of economics, the Journal of Public Economics (Chetty et al., 2014).

The authors started their experiment by randomly dividing the reviewers of the journal into four distinct groups:

1. a control group, with 45 days for the referee reports’ submission;

2. a first treatment group, subject to a shorter delivery time of 28 days;

3. a second treatment group, also subject to a 28-day term, but eligible to receive a cash incentive of US$100 per referee report handed on time; and

4. a third treatment group, subject to a social incentive, in which the reviewers would have their review preparation hours published on the journal’s website at the end of the year.

When evaluating their experiment results, the authors discovered interesting patterns arising from their experimental data. First, they noticed a reduction in the median time of submission of referee reports from 48 to 36 days approximately. Second, the introduction of a monetary incentive of US$100 per referee report reduced median review hours by additional eight days. Third, the treatment group subject to the social incentive (that is, which had their names published on the journal's website) presented a reduction of median review time of 2.5 days. This kind of incentive had a more significant impact in the case of tenured professors when compared to untenured referees.

Monetary incentives matter (as well as social incentives)

The most exciting aspect of the study by Chetty et al. (2014) is the result related to their experiment’s incentives. Unlike previous studies, in which monetary incentives ended up generating adverse effects in terms of behaviour (Gneezy & Rustichini, 2000; Squazzoni, Bravo & Takács, 2013), there was a substantial improvement in the time taken to prepare anonymous referee reports. In general terms, these results draw attention to the importance of incentives for referee reports, given the differences between individual incentives and social benefits in an academic setting. A recent body of evidence-based behavioural experiments suggests that people may care more about symbolic incentives – such as respect and status – than monetary gains (Ellingsen & Johannesson, 2007). It would be interesting to evaluate how referees respond to social and monetary incentives in real-world settings, given the differences between intrinsic and extrinsic motivations for collective action situations (Bénabou & Tirole, 2006).

Writing referee reports is so essential to developing a knowledge field that actions and interventions directed at the reviewer level “were associated with improved review quality compared with traditional methods” (Gaudino et al., 2021). Chetty et al.’s (2014) study focused on the monetary and social incentives and identified those gains.

If all those shreds of evidence are not enough to justify why we should care about writing referee reports, then the simple fact is that imperfect as it might be, there seems to be no better alternative to the peer review process at present. Therefore, we must continue to strive to improve it (London, 2021). For Gerhardt et al. (2014, p. 13), “the academic paper publishing process is implicitly based on a contract of trust between publishers and reviewers, authors, publishers and readers, and also on transparency and strictness about the procedures used to review papers”.

In a nutshell, “[…] reviewing is a professional responsibility” (Treviño, 2008, p. 8), and we must strive for academic progress despite all the problems and difficulties along the process.

Note

1. Because of the new open science practices, some referee reports now have the reviewers’ identification side by side with the authors’. This is becoming more and more common in the so-called “pre-print” format.