The purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.
The authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.
The authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.
The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.
Although a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.
The framework presented in this paper has been developed as part of the research project LibRank – New Approaches to Relevance Ranking in Library Information Systems funded by the German Research Foundation (DFG – Deutsche Forschungsgemeinschaft) from March 2014 until February 2016.
Behnert, C. and Lewandowski, D. (2017), "A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments", Journal of Documentation, Vol. 73 No. 3, pp. 509-527. https://doi.org/10.1108/JD-08-2016-0099
Emerald Publishing Limited
Copyright © 2017, Emerald Publishing Limited