To read the full version of this content please select one of the options below:

A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments

Christiane Behnert (Department of Information, Hamburg University of Applied Sciences, Hamburg, Germany)
Dirk Lewandowski (Department of Information, Hamburg University of Applied Sciences, Hamburg, Germany)

Journal of Documentation

ISSN: 0022-0418

Article publication date: 8 May 2017

Abstract

Purpose

The purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.

Design/methodology/approach

The authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.

Findings

The authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.

Practical implications

The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.

Originality/value

Although a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

Keywords

Acknowledgements

The framework presented in this paper has been developed as part of the research project LibRank – New Approaches to Relevance Ranking in Library Information Systems funded by the German Research Foundation (DFG – Deutsche Forschungsgemeinschaft) from March 2014 until February 2016.

Citation

Behnert, C. and Lewandowski, D. (2017), "A framework for designing retrieval effectiveness studies of library information systems using human relevance assessments", Journal of Documentation, Vol. 73 No. 3, pp. 509-527. https://doi.org/10.1108/JD-08-2016-0099

Publisher

:

Emerald Publishing Limited

Copyright © 2017, Emerald Publishing Limited