Library Trends: Assessing Digital Library Services Vol. 49 No. 2

Citation

Chowdhury, G.G. (2002), "Library Trends: Assessing Digital Library Services Vol. 49 No. 2", Journal of Documentation, Vol. 58 No. 2, pp. 229-232. https://doi.org/10.1108/jd.2002.58.2.229.3

This issue of Library Trends contains eight papers discussing the topical issue of digital library (DL) evaluation. Though DL research is almost a decade old now, to date very few works focusing on the evaluation aspects have been reported in the literature. One of the major reasons for this, as discussed by many authors in this issue, is that DLs are still at the experimental stage, and it may be a little premature to evaluate them. The other impediments for DL evaluation are that we hardly know what to evaluate and how to evaluate. DLs have been defined differently by researchers, and this makes the task of evaluation more complex. They are designed and developed with the specialisation of people from a number of fields, such as computer science, engineering, information science and so on, and they involve a host of social, economic, legal and management issues. Hence defining the standards and parameters for evaluation is a very complex task.

The eight papers that appear in this issue can be divided into three groups:

1. (1)

   papers that deal with the fundamental questions of DL evaluation including methodology, parameters etc.;

2. (2)

   papers that report the experience of evaluating some DLs; and

3. (3)

   papers that specifically deal with the evaluation of DL services, especially the reference services.

There are three papers that specifically deal with the fundamental questions of evaluation methodology etc. Saracevic discusses a number of issues related to the methodology and criteria for evaluation of DLs. Arguing that DLs are still at the stage of research and evolution, he suggests that the best practice standards against which performance levels can be measured are yet to be developed. He advocates a systems approach to evaluation that should aim to measure whether the DL, as a system, meets its stated objectives. Saracevic further suggests that in future DL evaluations may be conducted at several levels: from a broad social level to the level of content organisation, processing, retrieval, user interfaces, usability etc. He advocates a number of criteria for DL evaluation and divides them into three groups:

1. (1)

   traditional library criteria related to the collection, information content, use and standards;

2. (2)

   traditional information retrieval criteria related to relevance, satisfaction and output; and

3. (3)

   HCI (human computer interface) criteria related to usability, task, connectivity, design features, navigation etc.

Greenstein argues that a DL is “a networked online information space in which users can discover, locate, acquire, access and, increasingly, use information” (pp. 290‐1). According to him, one of the defining characteristics of DLs is that they facilitate access to the collections that they do not own, and hence they should be evaluated on how best they disclose, provide access to, and support the use of, their virtual collections. Greenstein emphasises the need for benchmarking standards for evaluation and reminds us that “digital libraries operate in a networked environment where they are both consumers and suppliers of digital collections and services” (p. 294).

Seadle suggests that people involved in a DL (project) must be understood before the DL itself can be assessed. He argues that anthropology can provide the initial understanding and the intellectual basis for choosing the design methodology and criteria. According to Seadle, most people are blends of micro‐cultures, which he defines as the units of meaning as small as professions, departments and interest groups. Drawing examples from the National Gallery of the Spoken World (NGSW) project (a Digital Library Initiative, second phase (DLI‐2) project), Seadle shows that a given DL project may consist of several micro‐cultures and a useful evaluation of DL services needs to include “an understanding of the nuances of the meaning and connotation, implementation and limitation, for a wide range of vocabulary across the many micro‐cultures involved” (p. 384). He further argues that “the precursor to developing a survey instrument, or selecting a survey population, or choosing the members of a focus group should involve an analysis of the project itself” (p. 384).

Borgman et al. report an overview of the evaluation project related to the development and deployment of ADEPT (Alexandria Digital Earth Prototype), a DLI‐2 project that is designed to “provide instructors and students with the means to discover, manipulate, and display dynamic geographical processes” (p. 229). This paper does not discuss the evaluation of a DL per se, rather it aims to assess the impact of the DL on undergraduate education. It specifically discusses the research questions, research design and the preliminary observations from the first year of the five‐year ADEPT project.

Marchionini advocates that the ultimate goal of DL evaluation is to assess the impact of DLs on a patron’s life and the larger social milieu. He defines evaluation as “a research process that aims to understand the meaning of some phenomenon situated in a context and the changes that take place as the phenomenon and the context interact” (p. 311). He reports the ongoing evaluation of the Perseus Digital Library (PDL) that is a DL of resources for the study of the ancient world. The evaluation primarily focuses on educational evaluation, i.e. how the electronic resources influence the educational context. Marchionini comments that “giving people control over how they access and use the DL satisfies a broader range of users and gives rise to wider ranges of application” (p. 330).

Gorman et al. focus on the information‐seeking patterns of users in a DL environment. They describe their observations based on a DLI‐2 project, “Tracking footprints in a medical information space: computer scientists‐physician collaborative study of expert problem solvers”. They suggest that experts create and use bundles – organised and highly selective collections of information – for solving problems. They conclude that “in the age of digital libraries computer‐based tools for creating and managing bundles may be needed as the information in these settings is increasingly represented in digital collections which promise to be much larger, more complex, more diverse, and more difficult to explore and manipulate” (p. 287).

Carter and Janes comment that providing a quality reference service without the needs and involvement of human intermediaries is one of the greatest challenges of DLs. They report a study that analysed the log of over 3,000 reference questions asked of the Internet Public Library (IPL), which is one of the most widely used reference services on the Web. This study has identified different categories of reference questions, such as common questions, quick questions, regular questions and unanswered questions, which were handled by IPL. The authors suggest that by carefully designing the question intake form, such a study can also explore a number of issues, such as user satisfaction, librarians’ attitudes, performance level over a period of time, and comparisons with other online reference services.

Peters comments that provision and evaluation of public services, such as reference services, have received little attention over the first decade of DL research. He suggests that for evaluation of online reference services it may be useful to concentrate on “meta assessment”, which he defines as the deliberate examination of the elements, basic conditions and needs of a thing – service, event, system and so on – that transcend particular instantiations of that thing. He comments that we need to assess several variable components of online reference services, and that a combination of controlled‐environment and real‐life research projects will be needed.

These papers, as the editor himself mentions, raise more questions than answers related to DL evaluation. Nevertheless, they not only raise a number of important issues related to the evaluation of DLs, but also stimulate thinking and thus pave the way for further research. This volume of Library Trends is definitely a valuable document for all DL researchers, and may be a good starting point for future DL evaluation experiments.