Search results

The main purpose of this study is to evaluate the effectiveness of relevance ranking on Google by comparing the system's assessment of relevance with the users' views. The research aims to find out whether the presumably objective relevance ranking of Google based on the PageRank and some other factors in fact matches users' subjective judgments of relevance.

This research investigated the relevance ranking of Google's retrieved results using 34 searches conducted by users in real search sessions. The results pages 1‐4 (i.e. the first 40 results) were examined by the users to identify relevant documents. Based on these data the frequency of relevant documents according to the appearance order of retrieved documents in the first four results pages was calculated. The four results pages were also compared in terms of precision.

In 50 per cent and 47.06 per cent of the searches the documents ranked 5th and 1st, (i.e. from the first pages of the retrieved results) respectively, were most relevant according to the users' viewpoints. Yet even in the fourth results pages there were three documents that were judged most relevant by the users in more than 40 per cent of the searches. There were no significant differences between the precision of the four results pages except between pages 1 and 3.

The results will help users of search engines, especially Google, to decide how many pages of the retrieved results to examine.

Search engine design will benefit from the results of this study as it experimentally evaluates the effectiveness of Google's relevance ranking.

One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements.

In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated “wisdom of crowds” judgements were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties.

It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings.

The proposed “wisdom of crowds”-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine’s ranking over time to changes in users preferences.

This is a first study that applies the notion of “wisdom of crowds” to examine an under-explored in the literature phenomenon of “change in time” in user evaluation of relevance.

In vol. 6, 1976, of Advances in Librarianship, I published a review about relevance under the same title, without, of course, “Part I” in the title (Saracevic, 1976). [A substantively similar article was published in the Journal of the American Society for Information Science (Saracevic, 1975)]. I did not plan then to have another related review 30 years later—but things happen. The 1976 work “attempted to trace the evolution of thinking on relevance, a key notion in information science, [and] to provide a framework within which the widely dissonant ideas on relevance might be interpreted and related to one another” (ibid.: 338).

The purpose of this paper is to demonstrate how to apply traditional information retrieval (IR) evaluation methods based on standards from the Text REtrieval Conference and web search evaluation to all types of modern library information systems (LISs) including online public access catalogues, discovery systems, and digital libraries that provide web search features to gather information from heterogeneous sources.

The authors apply conventional procedures from IR evaluation to the LIS context considering the specific characteristics of modern library materials.

The authors introduce a framework consisting of five parts: search queries, search results, assessors, testing, and data analysis. The authors show how to deal with comparability problems resulting from diverse document types, e.g., electronic articles vs printed monographs and what issues need to be considered for retrieval tests in the library context.

The framework can be used as a guideline for conducting retrieval effectiveness studies in the library context.

Although a considerable amount of research has been done on IR evaluation, and standards for conducting retrieval effectiveness studies do exist, to the authors’ knowledge this is the first attempt to provide a systematic framework for evaluating the retrieval effectiveness of twenty-first-century LISs. The authors demonstrate which issues must be considered and what decisions must be made by researchers prior to a retrieval test.

This paper aims to propose a new keyword search method on graph data to improve the relevance of search results and reduce duplication of content nodes in the answer trees obtained by previous approaches based on distinct root semantics. The previous approaches are restricted to find answer trees having different root nodes and thus often generate a result consisting of answer trees with low relevance to the query or duplicate content nodes. The method allows limited redundancy in the root nodes of top-k answer trees to produce more effective query results.

A measure for redundancy in a set of answer trees regarding their root nodes is defined, and according to the metric, a set of answer trees with limited root redundancy is proposed for the result of a keyword query on graph data. For efficient query processing, an index on the useful paths in the graph using inverted lists and a hash map is suggested. Then, based on the path index, a top-k query processing algorithm is presented to find most relevant and diverse answer trees given a maximum amount of root redundancy allowed for a set of answer trees.

The results of experiments using real graph datasets show that the proposed approach can produce effective query answers which are more diverse in the content nodes and more relevant to the query than the previous approach based on distinct root semantics.

This paper first takes redundancy in the root nodes of answer trees into account to improve the relevance and content nodes redundancy of query results over the previous distinct root semantics. It can satisfy the users’ various information need on a large and complex graph data using a keyword-based query.

How to provide the most useful papers for searchers is a key issue for academic search engines. A lot of research has been carried out to address this problem. However, when evaluating the effectiveness of an academic search engine, most of the previous investigations assume that the only concern of the user is the relevancy of the paper to the query. The authors believe that the usefulness of a paper is determined not only by its relevance to the query but also by other aspects including its publication age and impact in the research community. This is vital, especially when a large number of papers are relevant to the query.

This paper proposes a group of metrics to measure the usefulness of a ranked list of papers. When defining these metrics, three factors, including relevance, publication age and impact, are considered at the same time. To accommodate this, the authors propose a framework to rank papers by a combination of their relevance, publication age and impact scores.

The framework is evaluated with the ACL (Association for Computational Linguistics Anthology Network) dataset. It demonstrates that the proposed ranking algorithm is effective for improving usefulness when two or three aspects of academic papers are considered at the same time, while the relevance of the retrieved papers is slightly down compared with the relevance-only retrieval.

To the best of the authors’ knowledge, the proposed multi-objective academic search framework is the first of its kind that is proposed and evaluated with a group of new evaluation metrics.

The purpose of this paper is to improve users’ search results relevancy by manipulating their explicit feedback.

CoRRe – an explicit feedback model integrating three popular feedback, namely, Comment-Rating-Referral is proposed in this study. The model is further enhanced using case-based reasoning in retrieving the top-5 results. A search engine prototype was developed using Text REtrieval Conference as the document collection, and results were evaluated at three levels (i.e. top-5, 10 and 15). A user evaluation involving 28 students was administered, focussing on 20 queries.

Both Mean Average Precision and Normalized Discounted Cumulative Gain results indicate CoRRe to have the highest retrieval precisions at all the three levels compared to the other feedback models. Furthermore, independent t-tests showed the precision differences to be significant. Rating was found to be the most popular technique among the participants, producing the best precision compared to referral and comments.

The findings suggest that search retrieval relevance can be significantly improved when users’ explicit feedback are integrated, therefore web-based systems should find ways to manipulate users’ feedback to provide better recommendations or search results to the users.

The study is novel in the sense that users’ comment, rating and referral were taken into consideration to improve their overall search experience.

This paper aims to discover the effectiveness of Farsi-English query using ontology.

The present study is quasi-experimental. The sample consisted of 60 students and graduate and doctoral staff from Shiraz University and the Regional Center for Science and Technology. A researcher-made questionnaire was used to assess the level of English language proficiency of users, background knowledge and their level of satisfaction with search results before and after using ontology. Each user also evaluated the relevance of the top ten results on the Google search engine results page before and after using ontology.

The findings showed that the level of complexity of the task, the use of ontology, the interactive effect of the level of complexity of the task with the domain knowledge of the users, and the interactive effect of the level of complexity of the task with ontology, influence the effectiveness of retrieval results from the users' point of view. The results of the present study also showed that the level of complexity of the task, the use of ontology, and the interactive effect of the level of complexity of the task and the use of ontology, affect the level of user satisfaction.

The results of this research are significant in both theoretical and practical aspects. Theoretically, given the lack of research in which the interactive effect of the use of ontology has examined the level of complexity of tasks and domain knowledge of users, the present study can be considered as an attempt to improve information retrieval systems. From a practical point of view, the results of this research will help researchers and designers of information retrieval systems to understand that the use of ontologies can be used to retrieve information and improve the query and assess the needs of users and their satisfaction in this field, and ultimately, making the information retrieval process more effective.

This study intends to identify factors that affect relevance judgment of retrieved information as part of the 2007 TREC Legal track interactive task.

Data were gathered and analyzed from the participants of the 2007 TREC Legal track interactive task using a questionnaire which includes not only a list of 80 relevance factors identified in prior research, but also a space for expressing their thoughts on relevance judgment in the process.

This study finds that topicality remains a primary criterion, out of various options, for determining relevance, while specificity of the search request, task, or retrieved results also helps greatly in relevance judgment.

Relevance research should focus on the topicality and specificity of what is being evaluated as well as conducted in real environments.

If multiple relevance factors are presented to assessors, the total number in a list should be below ten to take account of the limited processing capacity of human beings' short‐term memory. Otherwise, the assessors might either completely ignore or inadequately consider some of the relevance factors when making judgment decisions.

This study presents a method for reducing the artificiality of relevance research design, an apparent limitation in many related studies. Specifically, relevance judgment was made in this research as part of the 2007 TREC Legal track interactive task rather than a study devised for the sake of it. The assessors also served as searchers so that their searching experience would facilitate their subsequent relevance judgments.

The purpose of this article is to look into relevance ranking and its importance in trying to bring some order to the deluge of results in response to a query.

A large‐scale analysis of detailed web logs of various search engines was performed. Sample tests were made on five to eight versions of MEDLINE, ERIC, and PsycINFO on hosts which have comparable versions of the databases and offer relevance ranking.

It was found that, for fairness, it must be ensured that the implementations are identical, they have the same retrospective coverage, the same MEDLINE/PubMed subsets, and (quasi) identical update.

The tests were made early September 2005. As databases are updated at different times, perfect synchronicity is not easy to achieve. When new records are added to the database, they may change the ranking of the test result set. Similarly, a small change in the fine‐tuning of the algorithm may yield different rank order positions of the same record the next time.

Brings together important research findings and suggests a topic for the next column.