Search results

This work aims to investigate the sensitivity of ranking performance with respect to the topic distribution of queries selected for ranking evaluation.

The authors reweight queries used in two TREC tasks to make them match three real background topic distributions, and show that the performance rankings of retrieval systems are quite different.

It is found that search engines tend to perform similarly on queries about the same topic; and search engine performance is sensitive to the topic distribution of queries used in evaluation.

Using experiments with multiple real‐world query logs, the paper demonstrates weaknesses in the current evaluation model of retrieval systems.

One of the under-explored aspects in the process of user information seeking behaviour is influence of time on relevance evaluation. It has been shown in previous studies that individual users might change their assessment of search results over time. It is also known that aggregated judgements of multiple individual users can lead to correct and reliable decisions; this phenomenon is known as the “wisdom of crowds”. The purpose of this paper is to examine whether aggregated judgements will be more stable and thus more reliable over time than individual user judgements.

In this study two simple measures are proposed to calculate the aggregated judgements of search results and compare their reliability and stability to individual user judgements. In addition, the aggregated “wisdom of crowds” judgements were used as a means to compare the differences between human assessments of search results and search engine’s rankings. A large-scale user study was conducted with 87 participants who evaluated two different queries and four diverse result sets twice, with an interval of two months. Two types of judgements were considered in this study: relevance on a four-point scale, and ranking on a ten-point scale without ties.

It was found that aggregated judgements are much more stable than individual user judgements, yet they are quite different from search engine rankings.

The proposed “wisdom of crowds”-based approach provides a reliable reference point for the evaluation of search engines. This is also important for exploring the need of personalisation and adapting search engine’s ranking over time to changes in users preferences.

This is a first study that applies the notion of “wisdom of crowds” to examine an under-explored in the literature phenomenon of “change in time” in user evaluation of relevance.

Relative performance evaluation (RPE) is a widely practiced employee appraisal process in the services industry. In a global delivery model, teams are spread across different geographical locations. The team members work on various tasks under the guidance of different managers and at times under more than one manager for performing the same task. Such complexities make RPE of the team members quite challenging. The paper proposes a methodical step-by-step approach to simplify the evaluation process without compromising on the rigour.

RPE has followed three different approaches. First is the traditional way, wherein evaluators had a common meeting to discuss and arrive at relative evaluation and ranking of members of the peer group employees. In the second, the number of evaluators and employees in a peer group were split in to 2 subgroups. The evaluators provided independent ratings and rankings. Simple mathematical tool then derived the combined ranking. In the third approach, each evaluator evaluated each employee in the peer group and provided the relative ranking for each employee. Again, mathematical tools provided the final ranking considering inputs from all evaluators. All the three evaluation approaches were analysed through an inter-rater agreement method.

All the three approaches for evaluation provided similar results giving confidence that less time-consuming methods could be adopted by evaluators without compromising on the rigour of the evaluation. The outcome of the exercise proved effective as the complaints reaching the ombudsmen reduced as compared to the earlier years. Considerable evaluation time was also saved. The study described in this paper is carried out in a non-unionized, Indian private sector services firm. Its effectiveness in other set ups is yet to be tested.

The research is carried out in the Indian Engineering services firm operating in the Knowledge based sector. Though study results are encouraging, the adaptability of methodology across different sectors and geographies is yet to be tested. More broad based studies are needed to evaluate suitability across firms and regions.

Relative evaluation exercise is challenging for evaluators. Although openness in evaluation is desired, it also makes evaluators uncomfortable in appearing to be taking sides or being opposing a candidate's ranking. The proposed approach brings in anonymity to each evaluator without scarifying individual evaluation.

The proposed methodology can be deployed across different services industries as the proposed methodology is business domain agnostic. It can be easily ported and tailored to align with an individual organization's evaluation philosophy. The suitability and effectiveness of the method can be studied under various types of firms like manufacturing, private, public, NGO, labour oriented, etc. As the proposed method reduces efforts, the stake holders can focus on understanding the relation between employee performance measurement, employee engagement, and long-term outcomes related to employee performance evaluation.

The proposed employee evaluation method leverages inter-rater reliability and agreement tool as a consensus approach to the relative performance ranking exercise. Such an approach to relative performance ranking is original as no prior studies with such an approach are found in the existing Literature.

The purposes of this study were to measure the relevance status of Index Islamicus, evaluate the semantic correlation between a query and documents and inquire the basis of its rank. Sorting the retrieved results from the most relevant to the least relevant is the common option of an information retrieval system. This sorting mechanism or relevance judgment is computed by measuring closeness of query with its documents.

Forming up 100 queries on Islamic History and Civilizations, with two indexing elements (keyword and concept), a laboratory experiment was generated on its first ten items of the rank. Throughout an experimental research design, the relevance status value formula was used to measure system-computed rank and compare it with mean average precision.

The results showed that the average status value of Index Islamicus’s ranking on relevance criterion was 18 per cent effective in terms of retrieving precise documents. Despite the main focus of this study being only on one subject domain and the items calculated were only 1,000, this small percentage of its ranking mechanism proved that semantic correlations between queries with subject domain did not achieve the satisfactory level.

Implication of this study could be a guideline for further research on ranking mechanism of other search engines because the limitation of this study was Index Islamicus being the only database, which was the focus of this study.

Throughout this study, Index Islamicus would be benefited knowing the status of its ranking mechanism as well as other databases can make further research on their own ranking method following this study.

Researchers and vendors of online databases can ensure their users a true platform of search engine with a proper ranking list.

Relevance status value model for Index Islamicus on Islamic History and Civilization that allows the system to rank documents according to the match between document and query and gives the idea of a better index. The model improves the system’s ranking mechanism, and promotes the use of semantic relationships. This research promotes the computation of relevance status value by domain for capturing subject-specific relevance criteria and semantic relationships.

Purpose — Ranking is a natural task for a search engine; a search engine result page is the most common example. This chapter aims at illustrating the motivations and the concepts of rank correlation in a practical way for the researchers active in the different domains of search engines.

Methodology/approach — To this end, this chapter provides a survey according to a topic-oriented basis of the search engine evaluation literature specifically devoted to or based on rank correlation; the chapter explains and illustrates how statistics is the only approach to rank correlation.

Findings/research limitations/implications — The chapter introduces the pros and cons of rank correlation measures through a light-weight formal description and a number of concrete examples to find the measure that better fit a context.

Practical implications — This chapter provides a blueprint for the application of rank correlation within scientific experimentation or item/service recommendation.

Social implications — Rank correlation analyses impact on the success or failure of a search engine in performing the tasks for which it has been designed and hence on the people's daily life activities.

Originality/value of paper — This chapter places rank correlation within a scientific research perspective and in particular connects to and complements documentation on search engine evaluation.

The purpose of this paper is to introduce a long‐time report evaluation project of the Institute for Ecological Economy Research (IÖW) and the business network “future – verantwortung unternehmen”.

On a regular basis, the IÖW/future‐Ranking evaluates the content and communicative quality of sustainability and CR reports of Germany's 150 largest companies. The project provides a platform where stakeholder expectations concerning corporate responsibility and transparency are synthesised into a comprehensive catalogue of reporting requirements. These criteria have to be met by the companies to reliably show sustainability commitment.

The evaluation of the reports not only makes the sustainability reporting practice of companies comparable and transparent to the public. The set of criteria itself also serves as a reporting framework for companies and thus enhances directional certainty about corporate contributions to sustainable development. It is itself subject to a periodic evaluation and adaptation by means of dialogue and feedback processes with representatives from industry, politics and NGOs.

The paper discusses the process of ranking and the possibilities for future developments in this.

The process described has had little academic exposure and so the paper provides an original insight into a practical subject which has academic interest.

This paper aims to find a way to personalize attraction recommendations for travelers. The research objective is to find a more accurate way to suggest new attractions to each traveler based on the opinions of other like-minded travelers and the traveler’s preferences.

To achieve the goal, developers have created a personalized system to generate attraction recommendations. The system considers an individual traveler’s preferences to construct a qualitative attraction ranking model. The new ranking model is the result of blending two processes: K-means clustering and the analytic hierarchy process (AHP).

The performance of the developed recommendation system has been assessed by measuring the accuracy and scalability of the ranking model of the system. The experimental results indicate that the ranking model always returns accurate results independent of the number of attractions and the number of travelers in each cluster. The ranking model has also proved to be scalable because the processing time is independent of the numbers of travelers. Additionally, the results reveal that the overall system usability is at a very satisfactory level.

The main theoretical implication is that integrating the processes of K-means and AHP techniques enables a new qualitative ranking model for personalized recommendations that deliver only high-quality attractions. However, the designed recommendation system has some limitations. First, it is necessary to manually update information about the new tourist attractions. Second, the overall response time depends on the internet bandwidth and latency.

This research contributes to the tourism business and individual travelers by introducing an accurate and scalable way to suggest new attractions to each traveler. The potential benefit includes possible increased revenue for travel agencies that offer personalized package tours and support individual travelers to make the final travel decisions. The designed system could also integrate with itinerary planning systems to plot out a journey that pinpoints what travelers will most enjoy.

This research proposes a design and implementation of a personalized recommendation system based on the qualitative attraction ranking model introduced in this article. The novel ranking model is designed and developed by integrating K-means and AHP techniques, which has proved to be accurate and scalable.

本研究主要探索如何建立个性化旅游胜地推荐模型。本研究通过分析旅游兴趣相似的游客意见和游客偏好选择, 建立一种更加准确推荐游客需要的旅游胜地方法。

为了达到研究目的, 本研究建立了一种个性化推荐旅游胜地的信息系统。其系统通过分析每个游客的旅游偏好来建设一种定性旅游胜地排名模型。这种新型模型主要通过结合以下两种分析算法：（1）K平均聚类算法（K-means clustering）（2）层次分析法（AHP）。

本研究建立的推荐信息系统经过了准确率和拓展性的测评。实验结果表明这种排名模型的准确率并不受旅游胜地多少和游客样本大小的影响。此外, 这种排名模型也具有拓展性, 因为算法时间并不受游客样本大小的影响。最后, 研究实验表明此排名模型客户体验性达到合格满意要求。

本研究的主要理论意义在于其结合了K平均聚类算法和层次分析法, 并建立了一种新型定性排名模型, 这种排名模型个性化地推荐更高质量的旅游胜地给游客。然而, 这种推荐信息系统有一些局限性。第一, 新旅游胜地的信息需要手动输入。第二, 整个系统的处理时间决定于网络带宽和延迟状况。

本研究的实践意义在于其建立了一种准确和具有拓展性的新型旅游胜地推荐模型。这种模型的潜在价值将有利于旅游机构提供定制化旅游套餐和帮助游客制定旅游计划。此外, 这种模型还可以结合旅游路线计划系统以制定更加使游客满意的旅游行程。

本研究推荐了一种基于定性旅游胜地排名模型的个性化旅游推荐模型。这种新型的排名模型结合K平均聚类算法和层次分析法, 实验证明这种模型更具准确性和拓展性。

The purpose of this paper is to use scholarly communication and citation data to rank the universities in a developing country perspective in Bangladesh.

This paper used publication and citation data from Scopus to rank the universities in Bangladesh. The number of papers published in each year during 2008-2018 by the select 33 universities, faculty per capita, the two-year citations and the number cited articles were used to arrive at a yearly weighted score and overall score for each university. The overall score was used to produce the ranking of the universities in Bangladesh.

The findings indicate that the ranking system can be implemented at the national level as an annual exercise to rank the higher education institutions in developing country perspective.

This is the first time an effort has been made to rank the universities in Bangladesh using publication and citation data. The findings of this paper could help policymakers, administrators and academicians to understand how the universities are performing in terms of scholarly activities.

This study distinguishes the academic influence of different papers published in journals of the same subject or field based on the modification of the journal impact factor.

Taking SSCI journals in library and information science (LIS) as the research object, the authors first explore the skewness degree of the citation distribution of journal articles. Then, we define the paper citation ratio as the weight of impact factor to modify the journal impact factor for the evaluation of papers, namely the weighted impact factor. The authors further explore the feasibility of the weighted impact factor in evaluating papers.

The research results show that different types of skewness exist in the citation distribution of journal papers. Particularly, 94% of journal paper citations are highly skewed, while the rest are moderately skewed. The weighted impact factor has a closer correlation with the citation frequency of papers than the journal impact factor. It resolves the issue that the journal impact factor tends to exaggerate the influence of low-cited papers in journals with high impact factors or weaken the influence of high-cited papers in journals with low impact factors.

The weighted impact factor is constructed based on the skewness of the citation distribution of journal articles. It provides a new method to distinguish the academic influence of different papers published in journals of the same subject or field, then avoids the situation that papers published in the same journal having the same academic impact.

This study reviews the methods found in the literature for the ranking of authors, identifies the pros and cons of these methods, discusses and compares these methods. The purpose of this paper is to study is to find the challenges and future directions of ranking of academic objects, especially authors, for future researchers.

This study reviews the methods found in the literature for the ranking of authors, classifies them into subcategories by studying and analyzing their way of achieving the objectives, discusses and compares them. The data sets used in the literature and the evaluation measures applicable in the domain are also presented.

The survey identifies the challenges involved in the field of ranking of authors and future directions.

To the best of the knowledge, this is the first survey that studies the author ranking problem in detail and classifies them according to their key functionalities, features and way of achieving the objective according to the requirement of the problem.