Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings

Maayan Zhitomirsky-Geffet (Department of Information Science, Bar-Ilan University, Ramat Gan, Israel)
Judit Bar-Ilan (Department of Information Science, Bar-Ilan University, Ramat Gan, Israel)
Mark Levene (Department of Computer Science, BirkBeck University of London, London, UK)

Aslib Journal of Information Management

ISSN: 2050-3806

Publication date: 18 July 2016



One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements.


In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated “wisdom of crowds” judgements were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties.


It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings.

Practical implications

The proposed “wisdom of crowds”-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine’s ranking over time to changes in users preferences.


This is a first study that applies the notion of “wisdom of crowds” to examine an under-explored in the literature phenomenon of “change in time” in user evaluation of relevance.



Zhitomirsky-Geffet, M., Bar-Ilan, J. and Levene, M. (2016), "Testing the stability of “wisdom of crowds” judgments of search results over time and their similarity with the search engine rankings", Aslib Journal of Information Management, Vol. 68 No. 4, pp. 407-427. https://doi.org/10.1108/AJIM-10-2015-0165

Download as .RIS



Emerald Group Publishing Limited

Copyright © 2016, Emerald Group Publishing Limited

Please note you might not have access to this content

You may be able to access this content by login via Shibboleth, Open Athens or with your Emerald account.
If you would like to contact us about accessing this content, click the button and fill out the form.
To rent this content from Deepdyve, please click the button.