To read this content please select one of the options below:

Expert judgments versus publication-based metrics: do the two methods produce identical results in measuring academic reputation?

Katerina Guba (European University at St Petersburg, Sankt-Peterburg, Russian Federation)
Angelika Tsivinskaya (European University at St Petersburg, Sankt-Peterburg, Russian Federation)

Journal of Documentation

ISSN: 0022-0418

Article publication date: 16 May 2022

Issue publication date: 10 January 2023

189

Abstract

Purpose

This study aims to assess the validity of citation metrics based on the disciplinary representative survey.

Design/methodology/approach

The present project compared citation rankings for individual scientists with expert judgments collected through a survey of 818 Russian sociologists. The Russian Index of Science Citation was used to construct the general population of 3,689 Russian sociologists, to whom the survey was sent by email. The regression analyses of bibliometric indicators and peer review scores for 723 names of scholars mentioned in the survey have been undertaken.

Findings

Findings suggest that scientometric indicators predict with significant accuracy the names of the most influential sociologists and those scholars who are not mentioned while they are less relevant for prediction names which received moderate attention in the survey.

Originality/value

This study contributes to the research on the validity of citation metrics by focusing on scientometric indicators, not limited to traditional metrics but including non-standard publication metrics and indicators of potential metric abuse. Besides, the study presents the national bibliometric data source that is especially important for non-Western higher education systems, less presented in the Web of Science or Scopus.

Keywords

Acknowledgements

The paper has been prepared with the support of the Russian Science Foundation, No. 21-18-00519.

Citation

Guba, K. and Tsivinskaya, A. (2023), "Expert judgments versus publication-based metrics: do the two methods produce identical results in measuring academic reputation?", Journal of Documentation, Vol. 79 No. 1, pp. 127-143. https://doi.org/10.1108/JD-02-2022-0039

Publisher

:

Emerald Publishing Limited

Copyright © 2022, Emerald Publishing Limited

Related articles