Evaluation of Digital Libraries: An Insight into Useful Applications and Methods

Citation

Dobreva, M. (2011), "Evaluation of Digital Libraries: An Insight into Useful Applications and Methods", Library Review, Vol. 60 No. 2, pp. 166-168. https://doi.org/10.1108/00242531111113131

Modern digital libraries have to address multiple diverging requirements and expectations of users and at the same time accommodate the rapidly changing technological novelties. The concepts of value, impact and user engagement of digital resources are discussed vigorously and in the current economic climate will be of even greater importance. Evaluation becomes a key activity in examining the extent to which a digital library provides content of high quality and services that match users' expectations and needs. The soundness of the digital library's underlying architecture is also an aspect subject to scrutiny. Evaluation is instrumental in comparing electronic resources: in this time of information, deluge helping the users to know which resource is better and in what particular way, is essential.

For these reasons, this edited collection on evaluation of digital libraries will definitely attract the interest of professionals working on digital resources. Practitioners from the digital library domain need sound consolidated guidance on evaluation methods and techniques because such guidance is currently dispersed between multiple publications, and although many provide an overview of evaluation (Fuhr et al., 2007), the content is generally insufficient to enable planning and implementation of a thorough step‐by‐step evaluation. To some extent, this is because evaluation of digital libraries is a domain which is still under development; it also is a vast domain addressing a range of machine‐ and human‐centric issues – and each of them is complex; for example, user evaluation is a domain in its own right with specific methodologies.

The editors have succeeded in accommodating a good range of evaluation issues and methods presented by some of the well‐known experts in the field, drawing connections between traditional library and digital library evaluation. The introductory chapter which pinpoints the challenges of evaluation of digital libraries is written by Tefko Saracevic. The rest of the contributions are structured in four parts, answering four key questions: to whom evaluation matters; what exactly is evaluated; why evaluation is necessary and how exactly evaluation should be performed.

Part 1, To whom it may concern, presents two perspectives – the user perspective which contextualises traditional library metrics as well as digital library ones in a chapter authored by Brinley Franklin, Martha Kyrllidou and Terry Plum, and the perspective of the funding agency written by Michael Khoo, David McArthur and Lee Zia. To some extent, this is a skewed view of the “whom” since users and funders are not the only stakeholders in the domain and points of view of technological developers and content providers also could be discussed.

Part 2 addresses, What to place under the evaluation lens, and includes contributions on usability (by Judy Jeng); users and evaluation (by Emmanouel Garoufallou, Rania Siatri and Richard Hartley); infrastructural issues for performance evaluation (by Maristella Agosti and Nicola Ferro) and deep log analysis for evaluating user behaviour based on the study of the British Library's learning web site BL Learning (by David Nicholas).

Behind the evaluation curtain, Part 3 offers chapters on design and evaluation (by Ann Blandford and David Bainbridge); assessment of outcomes (by Giannis Tsakonas and Christos Papatheodorou) and service quality in digital libraries (by Martha Kyrillidou, Colleen Cook and Yvonna Lincoln).

Part 4, How to conduct an evaluation activity, would be very useful if it synthesised some of the guidance from the previous parts and provided practical guidance on how to plan and conduct evaluation. Unfortunately, such a synthesis is not achieved – there are again individual chapters on planning specifically with logic models (by Michael Kjoo and Sarah Giersch), on qualitative (Maria Monopoli) and quantitative (Yin‐Leng Theng), but although these are of interest, the reader is not provided with really helpful and practical guidance on how to proceed effectively.

In general, the book helps the reader to better understand the multiple facets of evaluation, although it tends to focus more on the user‐centric side. It is indeed a good insight into “useful applications and methods”, as stated in its title. A major drawback is the very weak connection to digital libraries models, e.g. the DELOS DLRM (Candela et al., 2009) and 5S (Goncalves et al., 2004) – more could have been made of this. Evaluation of digital libraries inevitably addresses a combination of core concepts and a link to a formal model of a digital library would help to contextualise and explain exactly what is subject to evaluation and how such practice helps to further develop a particular digital resource.