A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments
Abstract
Purpose
The purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.
Design/methodology/approach
The authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.
Findings
The authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.
Practical implications
The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.
Originality/value
Although a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.
Keywords
Acknowledgements
The framework presented in this paper has been developed as part of the research project LibRank – New Approaches to Relevance Ranking in Library Information Systems funded by the German Research Foundation (DFG – Deutsche Forschungsgemeinschaft) from March 2014 until February 2016.
Citation
Behnert, C. and Lewandowski, D. (2017), "A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments", Journal of Documentation, Vol. 73 No. 3, pp. 509-527. https://doi.org/10.1108/JD-08-2016-0099
Publisher
:Emerald Publishing Limited
Copyright © 2017, Emerald Publishing Limited