Expert judgments versus publication-based metrics: do the two methods produce identical results in measuring academic reputation?
ISSN: 0022-0418
Article publication date: 16 May 2022
Issue publication date: 10 January 2023
Abstract
Purpose
This study aims to assess the validity of citation metrics based on the disciplinary representative survey.
Design/methodology/approach
The present project compared citation rankings for individual scientists with expert judgments collected through a survey of 818 Russian sociologists. The Russian Index of Science Citation was used to construct the general population of 3,689 Russian sociologists, to whom the survey was sent by email. The regression analyses of bibliometric indicators and peer review scores for 723 names of scholars mentioned in the survey have been undertaken.
Findings
Findings suggest that scientometric indicators predict with significant accuracy the names of the most influential sociologists and those scholars who are not mentioned while they are less relevant for prediction names which received moderate attention in the survey.
Originality/value
This study contributes to the research on the validity of citation metrics by focusing on scientometric indicators, not limited to traditional metrics but including non-standard publication metrics and indicators of potential metric abuse. Besides, the study presents the national bibliometric data source that is especially important for non-Western higher education systems, less presented in the Web of Science or Scopus.
Keywords
Acknowledgements
The paper has been prepared with the support of the Russian Science Foundation, No. 21-18-00519.
Citation
Guba, K. and Tsivinskaya, A. (2023), "Expert judgments versus publication-based metrics: do the two methods produce identical results in measuring academic reputation?", Journal of Documentation, Vol. 79 No. 1, pp. 127-143. https://doi.org/10.1108/JD-02-2022-0039
Publisher
:Emerald Publishing Limited
Copyright © 2022, Emerald Publishing Limited