Search results

The purpose of this study is to compare the evaluation of search result lists and documents, in particular evaluation criteria, elements, association between criteria and elements, pre/post and evaluation activities, and the time spent on evaluation.

The study analyzed the data collected from 31 general users through prequestionnaires, think aloud protocols and logs, and post questionnaires. Types of evaluation criteria, elements, associations between criteria and elements, evaluation activities and their associated pre/post activities, and time were analyzed based on open coding.

The study identifies the similarities and differences of list and document evaluation by analyzing 21 evaluation criteria applied, 13 evaluation elements examined, pre/post and evaluation activities performed and time spent. In addition, the authors also explored the time spent in evaluating lists and documents for different types of tasks.

This study helps researchers understand the nature of list and document evaluation. Additionally, this study connects elements that participants examined to criteria they applied, and further reveals problems associated with the lack of integration between list and document evaluation. The findings of this study suggest more elements, especially at list level, be available to support users applying their evaluation criteria. Integration of list and document evaluation and integration of pre, evaluation and post evaluation activities for the interface design is the absolute solution for effective evaluation.

This study fills a gap in current research in relation to the comparison of list and document evaluation.

The purpose of this paper is to propose a method to have more accurate results in comparing performance of the paired information retrieval (IR) systems with reference to the current method, which is based on the mean effectiveness scores of the systems across a set of identified topics/queries.

Based on the proposed approach, instead of the classic method of using a set of topic scores, the documents level scores are considered as the evaluation unit. These document scores are the defined document’s weight, which play the role of the mean average precision (MAP) score of the systems as a significance test’s statics. The experiments were conducted using the TREC 9 Web track collection.

The p-values generated through the two types of significance tests, namely the Student’s t-test and Mann-Whitney show that by using the document level scores as an evaluation unit, the difference between IR systems is more significant compared with utilizing topic scores.

Utilizing a suitable test collection is a primary prerequisite for IR systems comparative evaluation. However, in addition to reusable test collections, having an accurate statistical testing is a necessity for these evaluations. The findings of this study will assist IR researchers to evaluate their retrieval systems and algorithms more accurately.

A good recommender system helps users find items of interest on the web and can provide recommendations based on user preferences. In contrast to automatic technology‐generated recommender systems, this paper aims to use dynamic expert groups that are automatically formed to recommend domain‐specific documents for general users. In addition, it aims to test several effectiveness measures of rank order to determine if the top‐ranked lists recommended by the experts were reliable.

In the approach, expert groups evaluate web documents to provide a recommender system for general users. The authority and make‐up of the expert group are adjusted through user feedback. The system also uses various measures to gauge the difference between the opinions of experts and those of general users to improve the evaluation effectiveness.

The proposed system is efficient when there is major support from experts and general users. The recommender system is especially effective where there is a limited amount of evaluation data from general users.

This is an original study of how to effectively recommend web documents to users based on the opinions of human experts. Simulation results were provided to show the effectiveness of the dynamic expert group for recommender systems.

Hypertext documents can and do make large amounts of information readily accessible to ordinary readers, and production of such documents is understandably becoming more widespread. However, good intentions on the part of authors do not automatically result in the production of cost effective and acceptable documents. Evaluation is consequently a key issue — knowledge and use of appropriate evaluative methods helps ensure hypertext documents are correctly designed and presented. The results are increased usage and efficiency, and decreased maintenance costs. Drawing on practical, academic and commercial experience, this paper addresses some of the ways in which a hypertext document may be effectively evaluated.

For reasons of technical convenience, current retrieval algorithms based on probabilistic reasoning are derived from models that assume patrons evaluate documents using a two value relevance scale. This paper extends the theory by describing a model which includes a more general relevance scale. This model permits a re‐examination of the earlier theory as a special case of that developed here and leads to a more satisfying interpretation of the ranking principle of the earlier models.

One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements.

In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated “wisdom of crowds” judgements were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties.

It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings.

The proposed “wisdom of crowds”-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine’s ranking over time to changes in users preferences.

This is a first study that applies the notion of “wisdom of crowds” to examine an under-explored in the literature phenomenon of “change in time” in user evaluation of relevance.

The purpose of this paper is to explore what motivation strategies live-entertainment artistic directors (ADs) use to increase consistency in their employees’ performances.

The purpose of this qualitative case study was to explore the research question: what motivation strategies do live-entertainment ADs use to improve consistency in employee performance? Semistructured face-to-face interviews with artistic and senior ADs of a large international live-entertainment company’s US division participated in the study. In addition to the interviews, a further analysis of archival records of artists’ evaluations, and written company documents regarding performance evaluation to understand the ADs’ strategies were completed. Finally, self-reported interview data compared to AD evaluations of artists from randomly selected prior years verified the ADs practices.

The finding indicated ADs use multiple techniques geared at improving employee well-being and technical competence, thereby creating an environment conducive to the employees self-determining their consistent behavior in performance.

These findings may offer managers across multiple industries a variety of strategies and techniques to use to improve consistency for their workers.

This study is the one of few that studies manager influence on the motivation of those employees whose job is to entertain others regardless of the employee’s emotional state. From these findings, ADs may determine how to implement workplace safety improvements, expanding employee well-being, which in turn can improve performance consistency.

Learning to rank algorithms inherently faces many challenges. The most important challenges could be listed as high-dimensionality of the training data, the dynamic nature of Web information resources and lack of click-through data. High dimensionality of the training data affects effectiveness and efficiency of learning algorithms. Besides, most of learning to rank benchmark datasets do not include click-through data as a very rich source of information about the search behavior of users while dealing with the ranked lists of search results. To deal with these limitations, this paper aims to introduce a novel learning to rank algorithm by using a set of complex click-through features in a reinforcement learning (RL) model. These features are calculated from the existing click-through information in the data set or even from data sets without any explicit click-through information.

The proposed ranking algorithm (QRC-Rank) applies RL techniques on a set of calculated click-through features. QRC-Rank is as a two-steps process. In the first step, Transformation phase, a compact benchmark data set is created which contains a set of click-through features. These feature are calculated from the original click-through information available in the data set and constitute a compact representation of click-through information. To find most effective click-through feature, a number of scenarios are investigated. The second phase is Model-Generation, in which a RL model is built to rank the documents. This model is created by applying temporal difference learning methods such as Q-Learning and SARSA.

The proposed learning to rank method, QRC-rank, is evaluated on WCL2R and LETOR4.0 data sets. Experimental results demonstrate that QRC-Rank outperforms the state-of-the-art learning to rank methods such as SVMRank, RankBoost, ListNet and AdaRank based on the precision and normalized discount cumulative gain evaluation criteria. The use of the click-through features calculated from the training data set is a major contributor to the performance of the system.

In this paper, we have demonstrated the viability of the proposed features that provide a compact representation for the click through data in a learning to rank application. These compact click-through features are calculated from the original features of the learning to rank benchmark data set. In addition, a Markov Decision Process model is proposed for the learning to rank problem using RL, including the sets of states, actions, rewarding strategy and the transition function.

This paper reports the results of the information retrieval project PADOK‐II. This project, which began in November 1987, is being carried out by the Linguistic Information Science Group of the University of Regensburg (LIR) in cooperation with the German Patent Office (GPO) and is sponsored by the German Ministry for Research and Technology. The long‐term aim is to integrate artificial intelligence into information retrieval research without neglecting traditional information retrieval methodology. In PADOK‐II an information retrieval system is considered which indexes documents rather shallowly using free‐text or morphological components. A large‐scale retrieval test has been carried out, based on the German Patent Information System. Answers have been obtained to some 400 queries made by 10 users in simulated real‐life situations. These results have been used to attempt to answer the question: ‘How do the linguistically‐based functions of an indexing system contribute to its performance?’ As a spinoff of this test, the influence of document size and structure was studied with a view to identifying the most reasonable basic content for a German Patent Information System.

Nowadays the Internet offers a large amount of information to a wide range of users, making it difficult to deal with. The present work suggests the use of intelligent agents for the personalized filtering of Web pages. A set of autonomous, non‐mobile and adaptive agents was developed, aiming to satisfy the user’s need for information. The agents learn from the users’ feedback and attempt to produce better results over time. This work presents the system description and the promising results of tests performed in a simulated environment. The proposed system has proven to be a useful tool in reducing the amount of information with which the user has to deal.