Search results

To explore the status, interests, and intentions of peer reviewers and how editors enlist and muster these factors to enhance the prestige of a scholarly publication.

Case study: use of a 30‐year accumulation of editorial office records of one scholarly journal to analyze the contents of peer review comments and correspondence; direct quotes highlight key themes.

Peer reviewers labor to obtain more than the certification, authentication, and quality of individual works. The volume and variety of commentary generated by a double‐blind peer review process reveal concerns behind reviewer comments to authors and effects over time.

The study centers on one journal, Libraries & Culture, a publication committed to the specialized, interdisciplinary research about the history of libraries and the collection of cultural records.

The strategic nature of the administration and management of the invisible work of peer reviewers becomes more apparent. The interests and intentions of peer reviewers surface in commentary intended only for authors. Commentary relates to a variety of themes including personal interests, pedagogical and disciplinary objectives, field expansion agendas as well as the prestige of the publication. These themes suggest peer review as a potentially effective guiding mechanism for long‐term endeavors that benefit author, reviewer, and editor as interrelated players in arenas where distinction is at stake.

This study aimed to identify the characteristics of excellent peer reviewers by using Publons.com (an open and free online peer review website).

Reviewers of the clinical medicine field on Publons were selected as the sample (n = 1,864). A logistic regression model was employed to examine the data.

The results revealed that reviewers' verified reviews, verified editor records, and whether they were the Publons mentors had significant and positive associations with excellent peer reviewers, while their research performance (including the number of articles indexed by Web of Science (WOS), citations, H-index and high-cited researcher), genders, words per review, number of current/past editorial boards, whether they had experiences of post-publication review on Publons and whether they were Publons academy graduates had no significant associations with excellent peer reviewers.

This study could help journals find excellent peer reviewers from free and open online platforms.

The peer review history for this article is available at: https://publons.com/publon/10.1108/OIR-11-2021-0604.

Peer review has been with humans for a long time. Its effective inception dates back to World War II resulting information overload, which imposed a quantitative and qualitative screening of publications. Peer review was beset by a number of accusations and critics largely from the biases and subjective aspects of the process including the secrecy in which the processes became standard. Advent of the Internet in the early 1990s provided a manner to open peer review up to make it more transparent, less iniquitous, and more objective. This chapter investigates whether this openness led to a more objective manner of judging scientific publications. Three sites are examined: Electronic Transactions on Artificial Intelligence (ETAI), Atmospheric Chemistry and Physics (ACP), and Faculty of 1000 (F1000). These sites practice open peer review wherein reviewers and authors and their reviews and rebuttals are available for all to see. The chapter examines the different steps taken to allow reviewers and authors to interact and how this allows for the entire community to participate. This new prepublication reviewing of papers has to some extent, alleviated the biases that were previously preponderant and, furthermore, seems to give positive results and feedback. Although recent, experiences seem to have elicited scientists’ acceptance because openness allows for a more objective and fair judgment of research and scholarship. Yet, it will undoubtedly lead to new questions which are examined in this chapter.

The aim of this study is to examine how reviewers for academic journals learn to carry out the task of peer review and the issues they face in doing this.

A total of 45 reviewers completed a questionnaire which asked about their experience in doing peer reviews, how they had learnt to do them, and the issues they faced in doing these reviews. Follow up emails were also sent to reviewers in order to seek further elaboration on the answers they had provided in the questionnaire.

Over half of the reviewers had learnt to do reviews by reading reviews of their own submissions to peer‐reviewed journals. Others had learnt to write reviews by just doing them; that is, by practice. The most challenging aspect for the reviewers was writing reviewers' reports that were critical but still constructive. There was no consensus on the most straightforward aspects of writing peer reviews.

The study has implications for reviewer development, proposing an experiential, “learning by doing” approach to the training of reviewers rather than a didactic, information transmission style one.

The study has implications for reviewer development, proposing an experiential, “learning by doing” approach to the training of reviewers rather than a didactic, information transmission style one.

The study provides insights into how reviewers learn to write peer reviews and the challenges they face in doing this. The paper also suggests strategies for improving reviewer development which can have benefits, especially for early career researchers.

The Internet provides researchers with exciting new opportunities for finding information and communicating with each other. However the process of peer review is something of a Cinderella in all this. Peer review in biomedical disciplines is still largely carried out using hard copy and the postal system even if the authors’ text files are used for the production of the paper or electronic journal. This article introduces one of the Electronic Libraries (eLib) projects, funded by the Joint Information Systems Committee (JISC). The project – Electronic Submission and Peer Review (ESPERE) – is examining the cultural and technical problems of implementing an electronic peer review process for biomedical academics and learned society publishers. The paper describes preliminary work in discovering the issues involved and describes interviews with seven learned society publishers, analysis of a questionnaire sent to 200 editorial board members and a focus group of five biomedical academics. Academics and learned publishers were enthusiastic about electronic peer review and the possibilities which it offers for a less costly, more streamlined and more effective process. Use of the Internet makes collaborative and interactive refereeing a practical option and allows academics from countries all over the world to take part.

The purpose of this paper is to report successful authors’ views about the editorial review processes of a set of 42 accounting journals. The two main objectives are: to enlighten editors and journal publishers in their quest to improve their journals’ editorial review processes and to inform prospective authors about the past experiences successful authors have had with the 42 accounting journals.

A Webmail survey was used to collect data about authors’ experiences with publishing in one of the 42 accounting journals. A total of 856 responses (40 per cent response rate) was received. Various statistical analyses were used to explore a range of editorial review process features, including the timeliness of editorial feedback, timeliness of publishing accepted manuscripts, quality of the feedback provided and performance of the editor.

Authors were found to be generally quite satisfied with the editorial review processes of the journals in which they published. There were, however, notable leaders and laggards observed among the 42 journal titles. The survey findings also revealed that many journals use the practice of basing their editorial decisions on the comments of a single reviewer. In fact, this practice is most prevalent among the journals that are commonly perceived as the field’s “top” journals. These and other editorial review results – for example, comparisons between journal-tiers, geographical locations of editorial review offices and journal specialties – are discussed.

This paper extends and moves well beyond Adler and Liyanarachchi (2011), by exploring such additional author perceptions of the editorial review process as the performance of journal editors, the use of multiple reviewers and reviewers’ reporting of the typical faults/weaknesses in the papers they read. Exposing to public scrutiny an academic discipline’s editorial review processes is quite common in some fields of research, most notably medicine. Doing so in the accounting discipline addresses a need that many of the respondents felt was highly necessary and long overdue. While authors will benefit from the paper’s insights, editors and publishers are expected to as well.

In editorial peer review systems of journals, one does not always accept the best papers. Due to different human perceptions, the evaluation of papers by peer review (for a journal) can be different from the impact that a paper has after its publication (measured by number of citations received) in this or another journal. This system (and corresponding problems) is similar to the information retrieval process in a documentary system. Also there, one retrieves not always the most relevant documents for a certain topic. This is so because the topic is described in the command language of the documentary system and this command does not always completely cover the “real topic” that one wants to describe. This paper aims to address this issue.

Based on this statement classical information retrieval evaluation techniques were applied to the evaluation of peer review systems. Basic in such an information retrieval evaluation are the notions of precision and recall and the precision‐recall‐curve. Such notions are introduced here for the evaluation of peer review systems.

The analogues of precision and recall are defined and their curve constructed based on peer review data from the journal Angewandte Chemie – International Edition and on citation impact data of accepted papers by this journal or rejected but published elsewhere papers. It is concluded that, due to the imperfect peer review process (based on human evaluation), if we want to publish a high amount of qualified papers (the ones we seek), several non‐qualified papers should also be accepted.

The authors conclude that, due to the imperfect peer review process (based on human evaluation), if we want to publish a high amount of qualified papers (the ones we seek), one will also accept several non‐qualified papers.

The study aims to provide an analytical overview of current innovations in peer review and their potential impacts on scholarly communication.

The authors created a survey that was disseminated among publishers, academic journal editors and other organizations in the scholarly communication ecosystem, resulting in a data set of 95 self-defined innovations. The authors ordered the material using a taxonomy that compares innovation projects according to five dimensions. For example, what is the object of review? How are reviewers recruited, and does the innovation entail specific review foci?

Peer review innovations partly pull in mutually opposed directions. Several initiatives aim to make peer review more efficient and less costly, while other initiatives aim to promote its rigor, which is likely to increase costs; innovations based on a singular notion of “good scientific practice” are at odds with more pluralistic understandings of scientific quality; and the idea of transparency in peer review is the antithesis to the notion that objectivity requires anonymization. These fault lines suggest a need for better coordination.

This paper presents original data that were analyzed using a novel, inductively developed, taxonomy. Contrary to earlier research, the authors do not attempt to gauge the extent to which peer review innovations increase the “reliability” or “quality” of reviews (as defined according to often implicit normative criteria), nor are they trying to measure the uptake of innovations in the routines of academic journals. Instead, they focus on peer review innovation activities as a distinct object of analysis.

Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research's open peer review publishing platform.

PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings.

PeerJudge can predict F1000Research judgements from negative evaluations in reviewers' comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision.

PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments.

The purpose of this paper is to consider various strengths and weaknesses of the academic review process with an emphasis on the effect the process has on the relevance of business journals, particularly in the marketing literature.

The authors not only highlight some of the literature addressing the review process but also present insight and opinion largely based on decades of experience editing, reviewing, writing and publishing.

Reviewers can help develop research papers, but reviewers remain gatekeepers who, theoretically, protect journals from publishing research that would diminish the truthful body of knowledge within a field. However, many inefficiencies, some of which involve volition, allow one to question whether the review process as we know it best accomplishes that purpose.

Recognizing that reviewers affect journal prestige, the paper concludes with a number of ideas for improving the gate-keeping and developmental functions for academic articles.

Society should extract value from what appears in publicly circulated, academic, refereed journals. However, to the extent that the publication process interferes with objective dissemination of knowledge, that value is diminished and perhaps even absent.

The paper intends to stimulate frank conversation about the Academy’s refereed publication process and factors that tend to interfere with its function.