To read this content please select one of the options below:

Automatically detecting open academic review praise and criticism

Mike Thelwall (School of Mathematics and Computer Science, University of Wolverhampton, Wolverhampton, UK)
Eleanor-Rose Papas (F1000Research, F1000 Research Ltd., London, UK)
Zena Nyakoojo (F1000Research, F1000 Research Ltd., London, UK)
Liz Allen (F1000Research, F1000 Research Ltd., London, UK)
Verena Weigert (Jisc, Bristol, UK)

Online Information Review

ISSN: 1468-4527

Article publication date: 1 July 2020

Issue publication date: 20 August 2020

305

Abstract

Purpose

Peer reviewer evaluations of academic papers are known to be variable in content and overall judgements but are important academic publishing safeguards. This article introduces a sentiment analysis program, PeerJudge, to detect praise and criticism in peer evaluations. It is designed to support editorial management decisions and reviewers in the scholarly publishing process and for grant funding decision workflows. The initial version of PeerJudge is tailored for reviews from F1000Research's open peer review publishing platform.

Design/methodology/approach

PeerJudge uses a lexical sentiment analysis approach with a human-coded initial sentiment lexicon and machine learning adjustments and additions. It was built with an F1000Research development corpus and evaluated on a different F1000Research test corpus using reviewer ratings.

Findings

PeerJudge can predict F1000Research judgements from negative evaluations in reviewers' comments more accurately than baseline approaches, although not from positive reviewer comments, which seem to be largely unrelated to reviewer decisions. Within the F1000Research mode of post-publication peer review, the absence of any detected negative comments is a reliable indicator that an article will be ‘approved’, but the presence of moderately negative comments could lead to either an approved or approved with reservations decision.

Originality/value

PeerJudge is the first transparent AI approach to peer review sentiment detection. It may be used to identify anomalous reviews with text potentially not matching judgements for individual checks or systematic bias assessments.

Keywords

Acknowledgements

This project was funded by Jisc (www.jisc.ac.uk). Jisc suggested the idea for automatic peer review of open reviews, suggested F1000Research as the test dataset and helped to write this article.Competing interests: Three authors are currently employed by F1000Research Ltd. One author is employed by Jisc, which funded the project, enlisted advice from F1000Research and contributed to the write-up. The data analysis was conducted independently by a fifth author.

Citation

Thelwall, M., Papas, E.-R., Nyakoojo, Z., Allen, L. and Weigert, V. (2020), "Automatically detecting open academic review praise and criticism", Online Information Review, Vol. 44 No. 5, pp. 1057-1076. https://doi.org/10.1108/OIR-11-2019-0347

Publisher

:

Emerald Publishing Limited

Copyright © 2020, Emerald Publishing Limited

Related articles