Search results

1 – 10 of over 7000
To view the access options for this content please click here
Article
Publication date: 1 February 2000

Pia Borlund

This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems…

Downloads
1799

Abstract

This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems, the aim of which is to facilitate evaluation of IIR systems in a way which is as close as possible to realistic IR processes. The experimental setting consists of three components: (1) the involvement of potential users as test persons; (2) the application of dynamic and individual information needs; and (3) the use of multidimensional and dynamic relevance judgements. Hidden under the information need component is the essential central sub‐component, the simulated work task situation, the tool that triggers the (simulated) dynamic information needs. This paper also reports on the empirical findings of the metaevaluation of the application of this sub‐component, the purpose of which is to discover whether the application of simulated work task situations to future evaluation of IIR systems can be recommended. Investigations are carried out to determine whether any search behavioural differences exist between test persons‘ treatment of their own real information needs versus simulated information needs. The hypothesis is that if no difference exists one can correctly substitute real information needs with simulated information needs through the application of simulated work task situations. The empirical results of the meta‐evaluation provide positive evidence for the application of simulated work task situations to the evaluation of IIR systems. The results also indicate that tailoring work task situations to the group of test persons is important in motivating them. Furthermore, the results of the evaluation show that different versions of semantic openness of the simulated situations make no difference to the test persons’ search treatment.

Details

Journal of Documentation, vol. 56 no. 1
Type: Research Article
ISSN: 0022-0418

Keywords

To view the access options for this content please click here
Article
Publication date: 9 May 2016

Pia Borlund

The purpose of this paper is to report a study of how the test instrument of a simulated work task situation is used in empirical evaluations of interactive information…

Downloads
1353

Abstract

Purpose

The purpose of this paper is to report a study of how the test instrument of a simulated work task situation is used in empirical evaluations of interactive information retrieval (IIR) and reported in the research literature. In particular, the author is interested to learn whether the requirements of how to employ simulated work task situations are followed, and whether these requirements call for further highlighting and refinement.

Design/methodology/approach

In order to study how simulated work task situations are used, the research literature in question is identified. This is done partly via citation analysis by use of Web of Science®, and partly by systematic search of online repositories. On this basis, 67 individual publications were identified and they constitute the sample of analysis.

Findings

The analysis reveals a need for clarifications of how to use simulated work task situations in IIR evaluations. In particular, with respect to the design and creation of realistic simulated work task situations. There is a lack of tailoring of the simulated work task situations to the test participants. Likewise, the requirement to include the test participants’ personal information needs is neglected. Further, there is a need to add and emphasise a requirement to depict the used simulated work task situations when reporting the IIR studies.

Research limitations/implications

Insight about the use of simulated work task situations has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies.

Originality/value

Simulated work task situations are widely used in IIR studies, and the present study is the first comprehensive study of the intended and unintended use of this test instrument since its introduction in the late 1990’s. The paper addresses the need to carefully design and tailor simulated work task situations to suit the test participants in order to obtain the intended authentic and realistic IIR under study.

Details

Journal of Documentation, vol. 72 no. 3
Type: Research Article
ISSN: 0022-0418

Keywords

To view the access options for this content please click here
Article
Publication date: 1 August 1997

Pia Borlund and Peter Ingwersen

The paper describes the ideas and assumptions underlying the development of a new method for the evaluation and testing of interactive information retrieval (IR) systems…

Downloads
1767

Abstract

The paper describes the ideas and assumptions underlying the development of a new method for the evaluation and testing of interactive information retrieval (IR) systems, and reports on the initial tests of the proposed method. The method is designed to collect different types of empirical data, i.e. cognitive data as well as traditional systems performance data. The method is based on the novel concept of a ‘simulated work task situation’ or scenario and the involvement of real end users. The method is also based on a mixture of simulated and real information needs, and involves a group of test persons as well as assessments made by individual panel members. The relevance assessments are made with reference to the concepts of topical as well as situational relevance. The method takes into account the dynamic nature of information needs which are assumed to develop over time for the same user, a variability which is presumed to be strongly connected to the processes of relevance assessment.

Details

Journal of Documentation, vol. 53 no. 3
Type: Research Article
ISSN: 0022-0418

Keywords

To view the access options for this content please click here
Article
Publication date: 14 March 2016

Melanie Landvad Clemmensen and Pia Borlund

The purpose of this paper is to report a study of order effect in interactive information retrieval (IIR) studies. The phenomenon of order effect is well-known, and it is…

Abstract

Purpose

The purpose of this paper is to report a study of order effect in interactive information retrieval (IIR) studies. The phenomenon of order effect is well-known, and it is the main reason why searches are permuted (counter-balanced) between test participants in IIR studies. However, the phenomenon is not yet fully understood or investigated in relation to IIR; hence the objective is to increase the knowledge of this phenomenon in the context of IIR as it has implications for test design of IIR studies.

Design/methodology/approach

Order effect is studied via partly a literature review and partly an empirical IIR study. The empirical IIR study is designed as a classic between-groups design. The IIR search behaviour was logged and complementary post-search interviews were conducted.

Findings

The order effect between groups and within search tasks were measured against nine classic IIR performance parameters of search interaction behaviour. Order effect is seen with respect to three performance parameters (website changes, visit of webpages, and formulation of queries) shown by an increase in activity on the last performed search. Further the theories with respect to motivation, fatigue, and the good-subject effect shed light on how and why order effect may affect test participants’ IR system interaction and search behaviour.

Research limitations/implications

Insight about order effect has implications for test design of IIR studies and hence the knowledge base generated on the basis of such studies. Due to the limited sample of 20 test participants (Library and Information Science (LIS) students) inference statistics is not applicable; hence conclusions can be drawn from this sample of test participants only.

Originality/value

Only few studies in LIS focus on order effect and none from the perspective of IIR.

Details

Journal of Documentation, vol. 72 no. 2
Type: Research Article
ISSN: 0022-0418

Keywords

To view the access options for this content please click here
Article
Publication date: 6 March 2020

Mahdi Zeynali Tazehkandi and Mohsen Nowkarizi

The purpose was to evaluate the effectiveness of Google (as an international search engine) as well as of Parsijoo, Rismoon, and Yooz (as Persian search engines).

Abstract

Purpose

The purpose was to evaluate the effectiveness of Google (as an international search engine) as well as of Parsijoo, Rismoon, and Yooz (as Persian search engines).

Design/methodology/approach

In this research, Google search engine as an international search engine, and three local ones, Parsijoo, Rismoon, and Yooz, were selected for evaluation. Likewise, 32 subject headings were selected from the Persian Subject Headings List, and then simulated work tasks were assigned based on them. A total of 192 students from Ferdowsi University of Mashhad were asked to search for the information needed for simulated work tasks in the selected search engines, and then to copy the relevant website URLs in the search form.

Findings

The findings indicated that Google, Parsijoo, Rismoon, and Yooz had a significant difference in the precision, recall, and normalized discounted cumulative gain. There was also a significant difference in the effectiveness (average of precision, recall, and NDCG) of these four search engines in the retrieval of the Persian resources.

Practical implications

Users using an efficient search engine will attain more relevant documents, and Google search engine was more efficient in retrieving the Persian resources. It is recommended to use Google as it has a more efficient search.

Originality/value

In this research, for the first time, Google has been compared with local Persian search engines considering the new approach (simulated work tasks).

Details

Library Hi Tech, vol. 39 no. 1
Type: Research Article
ISSN: 0737-8831

Keywords

To view the access options for this content please click here
Book part
Publication date: 5 October 2011

David Elsweiler, Max L. Wilson and Brian Kirkegaard Lunn

Originally grounded in library and information science, the majority of information behaviour and information-seeking theories focus on task-based scenarios where users…

Abstract

Originally grounded in library and information science, the majority of information behaviour and information-seeking theories focus on task-based scenarios where users try to resolve information needs. While other theories exist, such as how people unexpectedly encounter information, for example, they are typically related back to tasks, motivated by work or personal goals. This chapter, however, focuses on casual-leisure scenarios that are typically motivated by hedonistic needs rather than information needs, where people engage in searching behaviours for pleasure rather than to find information. This chapter describes two studies on (1) television information behaviour and (2) the casual information behaviours described by users of Twitter. The first study focuses on a specific casual-leisure domain that is familiar to many, while the second indicates that our findings generalise to many other casual-leisure scenarios. The results of these two studies are then used to define an initial model of casual-leisure information behaviour, which highlights the key differences between casual-leisure scenarios and typical information behaviour theory. The chapter concludes by discussing how this new model of casual-leisure information behaviour challenges the way we design information systems, measure their value and consequently evaluate their support for users.

Details

New Directions in Information Behaviour
Type: Book
ISBN: 978-1-78052-171-8

To view the access options for this content please click here
Article
Publication date: 7 October 2014

Barbara Wildemuth, Luanne Freund and Elaine G. Toms

One core element of interactive information retrieval (IIR) experiments is the assignment of search tasks. The purpose of this paper is to provide an analytical review of…

Downloads
1881

Abstract

Purpose

One core element of interactive information retrieval (IIR) experiments is the assignment of search tasks. The purpose of this paper is to provide an analytical review of current practice in developing those search tasks to test, observe or control task complexity and difficulty.

Design/methodology/approach

Over 100 prior studies of IIR were examined in terms of how each defined task complexity and/or difficulty (or related concepts) and subsequently interpreted those concepts in the development of the assigned search tasks.

Findings

Search task complexity is found to include three dimensions: multiplicity of subtasks or steps, multiplicity of facets, and indeterminability. Search task difficulty is based on an interaction between the search task and the attributes of the searcher or the attributes of the search situation. The paper highlights the anomalies in our use of these two concepts, concluding with suggestions for future methodological research related to search task complexity and difficulty.

Originality/value

By analyzing and synthesizing current practices, this paper provides guidance for future experiments in IIR that involve these two constructs.

Details

Journal of Documentation, vol. 70 no. 6
Type: Research Article
ISSN: 0022-0418

Keywords

To view the access options for this content please click here
Article
Publication date: 19 September 2019

Sophie Rutter, Elaine G. Toms and Paul David Clough

To design effective task-responsive search systems, sufficient understanding of users’ tasks must be gained and their characteristics described. Although existing…

Abstract

Purpose

To design effective task-responsive search systems, sufficient understanding of users’ tasks must be gained and their characteristics described. Although existing multi-dimensional task schemes can be used to describe users’ search and work tasks, they do not take into account the information use environment (IUE) that contextualises the task. The paper aims to discuss these issues.

Design/methodology/approach

With a focus on English primary schools, in four stages a multi-dimensional task scheme was developed that distinguishes between task characteristics generic to all environments, and those that are specific to schools. In Stage 1, a provisional scheme was developed based upon the existing literature. In the next two stages, through interviews with teachers and observations of school children, the provisional scheme was populated and revised. In Stage 4, whether search tasks with the same information use can be distinguished by their characteristics was examined.

Findings

Ten generic characteristics were identified (nature of work task, search task originator, search task flexibility, search task doer, search task necessity, task output, search goal, stage in work task, resources and information use) and four characteristics specific to primary schools (curricular area, use in curricular area, planning and location). For the different information uses, some characteristics are more typical than others.

Practical implications

The resulting scheme, based on children’s real-life information seeking, should be used in the design and evaluation of search systems and digital libraries that support school children. More generally, the scheme can also be used in other environments.

Originality/value

This is the first study to develop a multi-dimensional task scheme that considers encompasses the IUE.

To view the access options for this content please click here
Article
Publication date: 21 November 2016

Jing Chen, Dan Wang, Quan Lu and Zeyuan Xu

With a mass of electronic multi-topic documents available, there is an increasing need for evaluating emerging analysis tools to help users and digital libraries analyze…

Abstract

Purpose

With a mass of electronic multi-topic documents available, there is an increasing need for evaluating emerging analysis tools to help users and digital libraries analyze these documents better. The purpose of this paper is to evaluate the effectiveness, efficiency and user satisfaction of THC-DAT, a within-document analysis tool, in reading a multi-topic document.

Design/methodology/approach

The authors reviewed related literature first, then performed a user-centered, comparative evaluation of two within-document analysis tools, THC-DAT and BOOKMARK. THC-DAT extracts a topic hierarchy tree using hierarchical latent Dirichlet allocation (hLDA) method and takes the context information into account. BOOKMARK provides similar functionality to the Table of Contents bookmarks in Adobe Reader. Three novel kinds of tasks were devised for participants to finish on two tools, with objective results to assess reading effectiveness and efficiency. And post-system questionnaires were employed to obtain participants’ subjective judgments about the tools.

Findings

The results confirm that THC-DAT is significantly more effective than BOOKMARK, while not inferior in efficiency. There is some evidence that suggests THC-DAT can slow down the process of approaching cognitive overload and improve users’ willingness to undertake difficult task. Based on qualitative data from questionnaires, the results indicate that users were more satisfied when using THC-DAT than BOOKMARK.

Practical implications

Adopting THC-DAT in digital libraries or electrical document reading systems contributes to promoting users’ reading performance, willingness to undertake difficult task and general satisfaction. Moreover, THC-DAT is of great value to addressing cognitive overload problem in the information retrieval field.

Originality/value

This paper evaluates a novel within-document analysis tool in analyzing a multi-topic document, and proved that this tool is superior to the benchmark in effectiveness and user satisfaction, and not inferior in efficiency.

Details

Library Hi Tech, vol. 34 no. 4
Type: Research Article
ISSN: 0737-8831

Keywords

To view the access options for this content please click here
Article
Publication date: 10 July 2007

Jenny Craven and Annika Nietzio

Purpose – The purpose of this paper is to describe research undertaken for the European Internet Accessibility Observatory (EIAO) project. It aims to demonstrate how…

Downloads
967

Abstract

Purpose – The purpose of this paper is to describe research undertaken for the European Internet Accessibility Observatory (EIAO) project. It aims to demonstrate how, using a task‐based approach, statistical measures can be applied to an initial assessment of a web site's accessibility, which could then be applied to further assessments to provide an evolving picture of the ongoing accessibility of a web site. Design/methodology/approachTask‐based assessments were used to assess the accessibility of web sites, using quantitative and qualitative analysis. The findings from this approach were mapped onto a probabilistic model, developed to assess the probability of an accessibility barrier relating to a specific feature or features of a web site. Findings – The paper finds that providing participants with a task instead of allowing them to randomly explore and evaluate a web site yielded more comparable results. For the EIAO project team, the benefit of the task‐based approach was that it allowed them to compare the user testing results with the results of the automated testing tool developed by the project. From the aggregation models included in the analysis, the most appropriate model and parameters were selected, and adjustments were made according to the comparison outcome. Research limitations/implications – Due to resource limitations and efficiency requirements, the assessments undertaken were limited to automatic evaluation, which could also be tested by the users. Therefore not all accessibility barriers in a web site could be identified. Despite this, it is felt that the outcome of the automatic analysis can be utilised as indicator for the overall accessibility of the web site. Originality/value – This paper provides a framework for web designers, commissioners, and policy makers to undertake a user focussed assessment of the accessibility of their web sites, which could be used in conjunction with other assessment methods.

Details

Performance Measurement and Metrics, vol. 8 no. 2
Type: Research Article
ISSN: 1467-8047

Keywords

1 – 10 of over 7000