Search results

The overall goal of science is to build a valid and reliable body of knowledge about the functioning of the world and how applying that knowledge can change it. As personnel and human resources management researchers, we aim to contribute to the respective bodies of knowledge to provide both employers and employees with a workable foundation to help with those problems they are confronted with. However, what research on research has consistently demonstrated is that the scientific endeavor possesses existential issues including a substantial lack of (a) solid theory, (b) replicability, (c) reproducibility, (d) proper and generalizable samples, (e) sufficient quality control (i.e., peer review), (f) robust and trustworthy statistical results, (g) availability of research, and (h) sufficient practical implications. In this chapter, we first sing a song of sorrow regarding the current state of the social sciences in general and personnel and human resources management specifically. Then, we investigate potential grievances that might have led to it (i.e., questionable research practices, misplaced incentives), only to end with a verse of hope by outlining an avenue for betterment (i.e., open science and policy changes at multiple levels).

This paper aims to quantify the quality of peer reviews, evaluate them from different perspectives and develop a model to predict the review quality. In addition, this paper investigates effective features to distinguish the reviews' quality. 

First, a fine-grained data set including peer review data, citations and review conformity scores was constructed. Second, metrics were proposed to evaluate the quality of peer reviews from three aspects. Third, five categories of features were proposed in terms of reviews, submissions and responses using natural language processing (NLP) techniques. Finally, different machine learning models were applied to predict the review quality, and feature analysis was performed to understand effective features.

The analysis results revealed that reviewers become more conservative and the review quality becomes worse over time in terms of these indicators. Among the three models, random forest model achieves the best performance on all three tasks. Sentiment polarity, review length, response length and readability are important factors that distinguish peer reviews’ quality, which can help meta-reviewers value more worthy reviews when making final decisions.

This study provides a new perspective for assessing review quality. Another originality of the research lies in the proposal of a novelty task that predict review quality. To address this task, a novel model was proposed which incorporated various of feature sets, thereby deepening the understanding of peer reviews.