Peer Review and the Production of Scholarly Knowledge: Automated Textual Analysis of Manuscripts Revised for Publication in Administrative Science Quarterly

Abstract

This chapter extends research on peer review by utilizing and assessing an emerging methodology: automated textual analysis. In a corpus of 38 papers successfully revised for publication in Administrative Science Quarterly, the authors found that measures based on exact wording (measured by plagiarism detection) and sentence similarity (measured by Word Mover’s Distance) performed well in capturing differences between original submissions and published papers. They identified the same overall pattern of revision that authors reported (intensive revision of Theory and Discussion sections, limited modification of Methods), and were strongly correlated with the turnover in references and hypotheses that occurred in the course of peer review. Automated textual analysis can usefully contribute to the study of manuscript change in peer review and other social scientific contexts, particularly as available textual corpora grow in size.

Citation

Strang, D. and Dokshin, F. (2019), "Peer Review and the Production of Scholarly Knowledge: Automated Textual Analysis of Manuscripts Revised for Publication in Administrative Science Quarterly", Zilber, T.B., Amis, J.M. and Mair, J. (Ed.) The Production of Managerial Knowledge and Organizational Theory: New Approaches to Writing, Producing and Consuming Theory (Research in the Sociology of Organizations, Vol. 59), Emerald Publishing Limited, Leeds, pp. 103-121. https://doi.org/10.1108/S0733-558X20190000059006