Search results
1 – 10 of over 69000Puts forward a framework for carrying out user evaluations ofexpert systems, which is based on user evaluations carried out in alarge British company over a period of three years…
Abstract
Puts forward a framework for carrying out user evaluations of expert systems, which is based on user evaluations carried out in a large British company over a period of three years. The aim of this framework is to ensure that the aims of the expert system designers and the users of the system are congruent. This in turn will make it more likely that the expert system will be a success. The framework covers aspects such as user type, updating knowledge, training, the value of the system and the human/computer interface.
Details
Keywords
Lynne Hall, Susan Jane Jones, Ruth Aylett, Marc Hall, Sarah Tazzyman, Ana Paiva and Lynne Humphries
This paper aims to briefly outline the seamless evaluation approach and its application during an evaluation of ORIENT, a serious game aimed at young adults.
Abstract
Purpose
This paper aims to briefly outline the seamless evaluation approach and its application during an evaluation of ORIENT, a serious game aimed at young adults.
Design/methodology/approach
In this paper, the authors detail a unobtrusive, embedded evaluation approach that occurs within the game context, adding value and entertainment to the player experience whilst accumulating useful data for the development team.
Findings
The key result from this study was that during the “seamless evaluation” approach, users were unaware that they had been participating in an evaluation, with instruments enhancing rather than detracting from the in‐role game experience.
Practical implications
This approach, seamless evaluation, was devised in response to player expectations, perspectives and requirements, recognising that in the evaluation of games the whole process of interaction including its evaluation must be enjoyable and fun for the user.
Originality/value
Through using seamless evaluation, the authors created an evaluation completely embedded within the “magic circle” of an in‐game experience that added value to the user experience whilst also yielding relevant results for the development team.
Details
Keywords
Nadjla Hariri and Yaghoub Norouzi
The present study aims to review the literature concerning Digital Libraries (DLs) and user interfaces in order to identify, determine, and suggest evaluation criteria for a DLs…
Abstract
Purpose
The present study aims to review the literature concerning Digital Libraries (DLs) and user interfaces in order to identify, determine, and suggest evaluation criteria for a DLs user interface. Accordingly, this study's objectives are threefold: explore which criteria exert a significant relationship with the DLs user interface; identify a set of criteria that appears to be useful for evaluating DLs user interface; and determine evaluation criteria that have more frequency and occurrence in the related texts reviewed.
Design/methodology/approach
To do it, first, identifying related texts was necessary. Consequently, keywords such as “DLs user interface evaluation”,” DLs user interfaces”, “DLs evaluation”, “DLs usability”, “user interface evaluation”, “DLs research”, “web sites user interface evaluation”, “user interface standards”, and the like have been searched in the web as well as in some leading databases including Emerald, Proquest, SagePub, ScienceDirect, LISA, ERIC, ACM, and Springer. After identifying and accessing more than 100 evaluative works and some related articles, theoretical and empirical, nearly 50 sources were chosen for final examination.
Findings
After reviewing related texts, three major categories are identified: user interface and DLs; DLs and usability; and other studies related to user interface; each one of three identified categories has its own subcategories. Additionally, 22 evaluation criteria for assessing DLs interface have been identified.
Research limitations/implications
The review does not claim to be comprehensive.
Practical implications
Hopefully, criteria such as feedback, ease of use, match between system and the real world, customization, user support, user workload, interaction, compatibility, visibility of system status, user experience, flexibility, and accessibility which have been less considered should be applied more in future, particularly user‐oriented, studies. Furthermore, it is expected that criteria mentioned here could help related bodies pay more attention to the evaluation of EISs, especially DLs interface.
Originality/value
It can be said that this study has contributed to the research into the evaluation of DL interface.
Details
Keywords
This conceptual piece presents a framework to aid libraries in gaining a more thorough and holistic understanding of their users and services. Through a presentation of the…
Abstract
This conceptual piece presents a framework to aid libraries in gaining a more thorough and holistic understanding of their users and services. Through a presentation of the history of library evaluation, a multidimensional matrix of measures is developed that demonstrates the relationship between the topics and perspectives of measurement. These measurements are then combined through evaluation criteria, and then different participants in the library system view those criteria for decision making. By implementing this framework for holistic measurement and cumulative evaluation, library evaluators can gain a more holistic knowledge of the library system and library administrators can be better informed for their decision‐making processes.
Details
Keywords
Sudatta Chowdhury, Monica Landoni and Forbes Gibb
The main objective of this paper is to review work on the usability and impact of digital libraries.
Abstract
Purpose
The main objective of this paper is to review work on the usability and impact of digital libraries.
Design/methodology/approach
Work on the usability and impact of digital libraries is reviewed. Specific studies on the usability and impact of digital libraries in specific domains are also discussed in order to identify general and specific usability and impact measures.
Findings
The usability studies reviewed in this paper show that a number of approaches have been used to assess usability. In addition to the technical aspects of digital library design (e.g. architecture, interfaces and search tools), there are a number of usability issues such as globalisation, localisation, language, culture issues, content and human information behaviour. Digital libraries should, however, be evaluated primarily with respect to their target users, applications and contexts.
Research limitations/implications
Although a digital library evaluation study may have several objectives, ranging from the evaluation of its design and architecture to the evaluation of its usability and its impact on the target users, this paper focuses on usability and impact.
Originality/value
This paper provides insights into the state‐of‐the art in relation to the usability and impact of digital libraries.
Details
Keywords
Rajat Kumar Behera, Pradip Kumar Bala and Rashmi Jain
Any business that opts to adopt a recommender engine (RE) for various potential benefits must choose from the candidate solutions, by matching to the task of interest and domain…
Abstract
Purpose
Any business that opts to adopt a recommender engine (RE) for various potential benefits must choose from the candidate solutions, by matching to the task of interest and domain. The purpose of this paper is to choose RE that fits best from a set of candidate solutions using rule-based automated machine learning (ML) approach. The objective is to draw trustworthy conclusion, which results in brand building, and establishing a reliable relation with customers and undeniably to grow the business.
Design/methodology/approach
An experimental quantitative research method was conducted in which the ML model was evaluated with diversified performance metrics and five RE algorithms by combining offline evaluation on historical and simulated movie data set, and the online evaluation on business-alike near-real-time data set to uncover the best-fitting RE.
Findings
The rule-based automated evaluation of RE has changed the testing landscape, with the removal of longer duration of manual testing and not being comprehensive. It leads to minimal manual effort with high-quality results and can possibly bring a new revolution in the testing practice to start a service line “Machine Learning Testing as a service” (MLTaaS) and the possibility of integrating with DevOps that can specifically help agile team to ship a fail-safe RE evaluation product targeting SaaS (software as a service) or cloud deployment.
Research limitations/implications
A small data set was considered for A/B phase study and was captured for ten movies from three theaters operating in a single location in India, and simulation phase study was captured for two movies from three theaters operating from the same location in India. The research was limited to Bollywood and Ollywood movies for A/B phase, and Ollywood movies for simulation phase.
Practical implications
The best-fitting RE facilitates the business to make personalized recommendations, long-term customer loyalty forecasting, predicting the company's future performance, introducing customers to new products/services and shaping customer's future preferences and behaviors.
Originality/value
The proposed rule-based ML approach named “2-stage locking evaluation” is self-learned, automated by design and largely produces time-bound conclusive result and improved decision-making process. It is the first of a kind to examine the business domain and task of interest. In each stage of the evaluation, low-performer REs are excluded which leads to time-optimized and cost-optimized solution. Additionally, the combination of offline and online evaluation methods offer benefits, such as improved quality with self-learning algorithm, faster time to decision-making by significantly reducing manual efforts with end-to-end test coverage, cognitive aiding for early feedback and unattended evaluation and traceability by identifying the missing test metrics coverage.
Details
Keywords
Chunnian Liu, Qi Tian and Xiaogang Zhu
This study aimed to analyze existing problems in the dissemination and management of emergency information on social media platforms, improve social media users' experience…
Abstract
Purpose
This study aimed to analyze existing problems in the dissemination and management of emergency information on social media platforms, improve social media users' experience regarding such information, increase the efficiency of emergency information dissemination and curb the spread of misinformation.
Design/methodology/approach
In this study, the emergency information quality on social media platforms was examined. Based on the evaluation principles of the quality of mature information, social media information characteristics and the rules of emergency information dissemination, combined with relevant academic research results, an index to evaluate the quality of emergency information on social media was constructed. In addition, the authors have introduced cloud theory as an information quality evaluation method and used social media users' emotional characteristics to assess information quality evaluation results. A comprehensive system for evaluating emergency information quality, including indexes, methods and detection strategies was established. Based on a comprehensive system, a case study was conducted on the forest fires in Sichuan Province and the African swine fever events as reported on the Zhihu platform. In accordance with the results of the case study, the authors expanded the research and introduced the emotional characteristics of social media users as an independent evaluation dimension to evaluate the quality of emergency information on social media.
Findings
The comprehensive system's effectiveness was verified through the case study. Further, it was found that users' emotional characteristics (reflected in their information behavior) are inconsistent with their evaluation of websites' information quality regarding major emergencies. Integrating users' emotional characteristics into the information evaluation system can enhance its effectiveness following major emergencies.
Originality/value
First, an evaluation index system of emergency information quality on social media about major emergencies was offered. Unlike the commonly available index system for information quality evaluation, this proposed evaluation index system not only accounted for the characteristics of social media, such as massive disordered information, multiple information sources and rapid dissemination, but also for the characteristics of emergency events, such as variability and the absence of precursors. This proposed evaluation index system enhances the pertinence of the information quality evaluation and compensates for the shortcoming that the current research only focuses on evaluating social media information quality in a broad context, but pays insufficient attention to major emergencies. Second, cloud theory was introduced as a method to evaluate the emergency information quality found on social media. Existing research has primarily included the use of traditional statistical methods, which cannot transform numerical values into qualitative concepts effectively. Various indeterminate factors inevitably affect the quality of emergency information on social media platforms, and the traditional methods cannot eliminate this uncertainty in the evaluation process. The method to assess emergency information quality based on cloud theory can effectively compensate for the gaps in the research and improve the accuracy of information quality assessment. Third, the inspection and the dynamic adjustment of assessment results are absent in the research on information quality assessment, and the research has relied principally on the information users' evaluation and has paid insufficient attention to their attitudes and behaviors toward information. Therefore, the authors incorporated users' emotional characteristics into the evaluation of emergency information quality on social media and used them to test the evaluation results so that the results of the information quality assessment not only include the users' explicit attitudes but also their implicit attitudes. This enhances the effectiveness of the information quality assessment system. Finally, through this case study, it was found that an inconsistency exists between user evaluation and user emotional characteristics after major emergencies. The reasons for this phenomenon were explained, and the necessity of integrating user emotional characteristics into information quality assessment was demonstrated. Based on this, the users' emotional characteristics were used as a separate evaluation dimension for assessing the quality of emergency information on social media. Compared with assessing the quality of general information, integrating the user's emotional characteristics into the evaluation index system can lead the evaluation results to include not only the users' cognitive evaluation but also their emotional experience, further enhancing their adaptability.
Details
Keywords
Huawei Zhu, Rungting Tu, Wenting Feng and Jiaojiao Xu
Extreme online reviews can have great impacts on consumers’ purchase decisions. The purpose of this paper is to investigate when users are more likely to provide extreme ratings…
Abstract
Purpose
Extreme online reviews can have great impacts on consumers’ purchase decisions. The purpose of this paper is to investigate when users are more likely to provide extreme ratings. The study draws inference from attitude certainty theory and proposes that review extremity is influenced by the interaction of evaluation duration and product/service types: for hedonic products/services, shorter evaluation duration can foster attitude certainty, leading to higher review extremity; in contrast, for utilitarian products/services, longer evaluation duration can increase attitude certainty, resulting in more extreme reviews.
Design/methodology/approach
Three studies were conducted to test the hypotheses: Study 1 is an empirical analysis of 3,000 reviews from an online retailing website; Studies 2 and 3 are two between-subject experiments.
Findings
Results from three studies confirm the hypotheses. Study 1 provides preliminary evidence on how review extremity varies in evaluations of different durations and product/service types. Results from Studies 2 and 3 show that for hedonic products/services, the shorter the evaluation duration, the more likely users are to give extreme ratings; however, for utilitarian products/service, the longer the evaluation duration, the more likely users are to give extreme reviews; and attitude certainty plays a mediating role between evaluation duration and review extremity.
Originality/value
Findings from this study provide understandings on when a fast rather than a slow evaluation can lead to more extreme reviews. The results also highlight the role of users’ attitude certainty in the underlying mechanism.
Details
Keywords
Vassilios P. Aggelidis and Prodromos D. Chatzoglou
It is widely accepted that the use of information and communication technology (ICT) in the healthcare sector offers great potential for improving the quality of services…
Abstract
Purpose
It is widely accepted that the use of information and communication technology (ICT) in the healthcare sector offers great potential for improving the quality of services provided, the efficiency and effectiveness of personnel, and also reducing organizational expenses. This paper seeks to examine various hospital information system (HIS) evaluation methods.
Design/methodology/approach
In this paper a comprehensive search of the literature concerning the evaluation of complex health information systems is conducted and used to generate a synthesis of the literature around evaluation efforts in this field. Three approaches for evaluating hospital information systems are presented – user satisfaction, usage, and economic evaluation.
Findings
The main results are that during the past decade, computers and information systems, as well as their resultant products, have pervaded hospitals worldwide. Unfortunately, methodologies to measure the various impacts of these systems have not evolved at the same pace. To summarize, measurement of users' satisfaction with information systems may be the most effective evaluation method in comparison with the rest of the methods presented.
Practical implications
The methodologies, taxonomies and concepts presented in this paper could benefit researchers and practitioners in the evaluation of HISs.
Originality/value
This review points out the need for more thorough evaluations of HISs that look at a wide range of factors that can affect the relative success or failure of these systems.
Details
Keywords
This paper aims to provide a holistic view of the current practice of chat reference evaluation and to suggest a framework that could help reference practitioners evaluate chat…
Abstract
Purpose
This paper aims to provide a holistic view of the current practice of chat reference evaluation and to suggest a framework that could help reference practitioners evaluate chat reference services in multiple contexts.
Design/methodology/approach
A thorough review of the literature on chat reference evaluation is conducted and the evaluation studies are grouped by their evaluative perspective and measures. Based on the literature review, a framework of perspectives and measures for chat reference evaluation is proposed.
Findings
Chat reference evaluation has incorporated a number of new elements that do not exist in desk reference evaluation. All the evaluation perspectives and measures reported in chat reference literature are incorporated into a final framework except for one variable – “cost‐effectiveness” – which lacks the support of empirical studies in the literature.
Practical implications
The analytical review of the literature provides a holistic view of how chat reference is being measured for its value to both libraries and users, thus furthering the professional understanding of chat reference performance in the library environs. The framework of evaluation perspectives and measures resulting from the literature review is applicable in multiple chat reference contexts and can be customized to serve different evaluation purposes. In turn, this framework gives chat reference evaluators a clear idea of what to look at and how.
Originality/value
This paper fills the need to provide reference practitioners with both a critical view of current chat reference practice, and a tool that could help them design and develop a chat reference evaluation project.
Details