Search results

The purpose of this paper is to better understand the dynamic nature of the relevance judgment process and the influence of work task on that process.

The empirical study reported here examined the information seeking behavior of a group of undergraduate college students assigned a set of research assignments (work tasks). Subjects recorded their selection of documents used for an assignment and the criteria used to judge those documents relevant. Statistical analysis was used to associate relevance judgments and the criteria used to make those judgments with work tasks.

Findings indicate a strong statistical association between work task and criteria used to judge relevance. Findings also include identification of specific criteria used to judge relevance and the relative importance of those criteria based on frequency of selection of criteria for a work task.

Findings provide additional insights into the dynamic nature of the relevance judgment process. Relevance judgment influences revealed in these findings in the form of criteria used to make relevance judgments further explicate the relevance judgment process and provide suggestions for the improvement of information retrieval systems and information literacy efforts.

Understanding the relevance judgment process is critical to understanding information behavior in general. Few studies have examined relevance criteria selections as part of the relevance judgment process and fewer still have studied these selections in relation to work tasks. A better understanding of this relationship is an essential part of understanding the dynamic nature of the relevance judgment process and its influences.

In a system-based approach, replicating the web would require large test collections, and judging the relevancy of all documents per topic in creating relevance judgment through human assessors is infeasible. Due to the large amount of documents that requires judgment, there are possible errors introduced by human assessors because of disagreements. The paper aims to discuss these issues.

This study explores exponential variation and document ranking methods that generate a reliable set of relevance judgments (pseudo relevance judgments) to reduce human efforts. These methods overcome problems with large amounts of documents for judgment while avoiding human disagreement errors during the judgment process. This study utilizes two key factors: number of occurrences of each document per topic from all the system runs; and document rankings to generate the alternate methods.

The effectiveness of the proposed method is evaluated using the correlation coefficient of ranked systems using mean average precision scores between the original Text REtrieval Conference (TREC) relevance judgments and pseudo relevance judgments. The results suggest that the proposed document ranking method with a pool depth of 100 could be a reliable alternative to reduce human effort and disagreement errors involved in generating TREC-like relevance judgments.

Simple methods proposed in this study show improvement in the correlation coefficient in generating alternate relevance judgment without human assessors while contributing to information retrieval evaluation.

This study intends to identify factors that affect relevance judgment of retrieved information as part of the 2007 TREC Legal track interactive task.

Data were gathered and analyzed from the participants of the 2007 TREC Legal track interactive task using a questionnaire which includes not only a list of 80 relevance factors identified in prior research, but also a space for expressing their thoughts on relevance judgment in the process.

This study finds that topicality remains a primary criterion, out of various options, for determining relevance, while specificity of the search request, task, or retrieved results also helps greatly in relevance judgment.

Relevance research should focus on the topicality and specificity of what is being evaluated as well as conducted in real environments.

If multiple relevance factors are presented to assessors, the total number in a list should be below ten to take account of the limited processing capacity of human beings' short‐term memory. Otherwise, the assessors might either completely ignore or inadequately consider some of the relevance factors when making judgment decisions.

This study presents a method for reducing the artificiality of relevance research design, an apparent limitation in many related studies. Specifically, relevance judgment was made in this research as part of the 2007 TREC Legal track interactive task rather than a study devised for the sake of it. The assessors also served as searchers so that their searching experience would facilitate their subsequent relevance judgments.

The purpose of this paper is to specify user‐defined relevance criteria by which people select hyperlinks and pages in web searching.

A quantitative and qualitative analysis was undertaken of talking aloud data from nine web searches conducted about self‐generated topics.

Altogether 18 different criteria for selecting hyperlinks and web pages were found. The selection is constituted, by two, intertwined processes: the relevance judgment of hyperlinks, and web pages by user‐defined criteria, and decision‐making concerning the acceptance or rejection of hyperlinks and web pages. The study focuses on the former process. Of the individual criteria, specificity, topicality, familiarity, and variety were used most frequently in relevance judgments. The study shows that despite the high number of individual criteria used in the judgments, a few criteria such as specificity and topicality tend to dominate. Searchers were less critical in the judgment of hyperlinks than deciding whether the activated web pages should be consulted in more detail.

The study is exploratory, drawing on a relatively low number of case searches.

The paper gives a detailed picture of the criteria used in the relevance judgments of hyperlinks and web pages. The study also discusses the specific nature of criteria used in web searching, as compared to those used in traditional online searching environments.

In vol. 6, 1976, of Advances in Librarianship, I published a review about relevance under the same title, without, of course, “Part I” in the title (Saracevic, 1976). [A substantively similar article was published in the Journal of the American Society for Information Science (Saracevic, 1975)]. I did not plan then to have another related review 30 years later—but things happen. The 1976 work “attempted to trace the evolution of thinking on relevance, a key notion in information science, [and] to provide a framework within which the widely dissonant ideas on relevance might be interpreted and related to one another” (ibid.: 338).

One of the most effective ways information retrieval (IR) systems including Web search engines can improve relevance performance is to provide their users with tools for facilitating query expansion. Search engines such as Google provide users with keyword suggest tools. This paper aims to investigate users’ criteria in relevance judgment regarding Google’s keywords suggest tool and to see how such keywords would lead to more relevant results from the viewpoint of users.

Through a mixed method approach, quantitative and qualitative data were collected from 60 postgraduate students at Ferdowsi University of Mashhad, Iran, using four different instruments (questionnaire, thinking aloud technique, query logs and interviews).

Among other criteria, the “relation between suggested keywords and the information need” (with the mean rate of 3.53 of four) was considered the most important by searchers in selecting suggested keywords for query expansion. Also, the “relation between suggested Keywords and the retrieved items” (with the mean rate of 3.62) was considered the second most important criterion in judging the relevance of the retrieved results. The participants agreed that the suggested keywords by Google improved the retrieval relevance. The content analysis of the participants’ aloud-thinking sessions and the interviews approved such findings.

This research makes a contribution to the need of designers of IR systems regarding the use of add words for query expansion. It also helps librarians how to instruct searchers with expanding their queries to retrieve more relevant results. Another contribution of the study is the identification of a number of new relevance judgment criteria for Web-based environments.

There is some evidence of a slackening of interest in the testing and evaluation of retrieval systems. The estimated significance of this problem also appears to have decreased markedly. A literature search reveals the existence of less than fifty papers published to date in 1966, most of which do not report experimental activity. This dearth of significant effort may be due to a number of factors: investigators engaged in testing and evaluation have exhausted their repertoire; difficulty in undertaking further effort and at the same time satisfying more stringent experimental design requirements; realization that there never was any theoretical basis to testing and evaluation in the first place and that some basic‐type research is necessary before beginning any new testing activity. It is disappointing to consider how little the work on evaluation has contributed to the improved design and optimization of information retrieval systems. Little agreement exists either with respect to the nature of the phenomena which we wish to subject to quantification (what we are testing) or as to the requisite methodology (bow we are to test).

Automated sentence‐level relevance and novelty detection would be of direct benefit to many information retrieval systems. However, the low level of agreement between human judges performing the task is an issue of concern. In previous approaches, annotators were asked to identify sentences in a document set that are relevant to a given topic, and then to eliminate sentences that do not provide novel information. This paper aims to explore a new approach in which relevance and novelty judgments are made within the context of specific, factual information needs, rather than with respect to a broad topic.

An experiment is conducted in which annotators perform the novelty detection task in both the topic‐focused and fact‐focused settings.

Higher levels of agreement between judges are found on the task of identifying relevant sentences in the fact‐focused approach. However, the new approach does not improve agreement on novelty judgments.

The analysis confirms the intuition that making sentence‐level relevance judgments is likely to be the more difficult of the two tasks in the novelty detection framework.

The purpose of this paper is to examine changes in relevance assessments, specifically the selection of relevance criteria by subjects as they move through the information search process.

The paper examines the relevance criteria choices of 39 subjects in relation to search stage. Subjects were assigned a specific search task in a controlled test. Statistics were collected and analyzed using descriptive statistics and the chi‐square goodness‐of‐fit tests.

The statistically significant findings identified a number of commonly reported relevance criteria, which varied over an information search process for relevant and partially relevant judgments. These results provide statistical confirmations of previous studies, and extend these findings identifying specific criteria for both relevant and partially relevant judgments.

The study only examines a short duration search process and since the convenience sample of subjects were from similar backgrounds and were assigned similar tasks, the study did not explicitly examine the impact of contextual factors such as user experience, background or task in relation to relevance criteria choices.

The paper has implications for the development of search systems which are adaptive and recognize the cognitive changes which occur during the information search process. Examining and identifying relevance criteria beyond topicality and the importance of those criteria to a user can help in the generation of better search queries.

The paper adds more rigorous statistical analysis to the study of relevance criteria and the information search process.

The purpose of this paper is to investigate how healthcare professionals search for and select the medical images they need within medical settings.

In total, 29 healthcare professionals participated in the study. Using a think‐aloud technique and face‐to‐face interviews, the authors asked participants to explain how they looked for medical images and how they judged the relevancy of retrieved images.

A total of 15 criteria were applied by participants when determining the relevance of medical images in relation to their information needs. Topicality was found to be the primary and most important criterion used by participants.

This study compares the relevance criteria used for medical images with those identified in the literature and highlights six additional criteria that have not been suggested in previous work.