Search results

The purpose of this paper is to explore uncertainty inherent in emotion recognition technologies and the consequences resulting from that phenomenon.

The paper is a general overview of the concept; however, it is based on a meta-analysis of multiple experimental and observational studies performed over the past couple of years.

The main finding of the paper might be summarized as follows: there is uncertainty inherent in emotion recognition technologies, and the phenomenon is not expressed enough, not addressed enough and unknown by the users of the technology.

Practical implications of the study are formulated as postulates for the developers, users and researchers dealing with the technologies of automatic emotion recognition.

As technologies that recognize emotions are becoming more and more common, and perhaps more decisions influencing people lives are to come in the next decades, the trustworthiness of the technology is important from a scientific, practical and ethical point of view.

Studying uncertainty of emotion recognition technologies is a novel approach and is not explored from such a broad perspective before.

Uncertainty is an under-respected issue when it comes to automatic assessment of human emotion by machines. The purpose of this paper is to highlight the existent approaches towards such measurement of uncertainty, and identify further research need.

The discussion is based on a literature review.

Technical solutions towards measurement of uncertainty in automatic emotion recognition (AER) exist but need to be extended to respect a range of so far underrepresented sources of uncertainty. These then need to be integrated into systems available to general users.

Not all sources of uncertainty in automatic emotion recognition (AER) including emotion representation and annotation can be touched upon in this communication.

AER systems shall be enhanced by more meaningful and complete information provision on the uncertainty underlying their estimates. Limitations of their applicability should be communicated to users.

Users of automatic emotion recognition technology will become aware of their limitations, potentially leading to a fairer usage in crucial application context.

There is no previous discussion including the technical view point on extended uncertainty measurement in automatic emotion recognition.

Although numerous signal modalities are available for emotion recognition, audio and visual modalities are the most common and predominant forms for human beings to express their emotional states in daily communication. Therefore, how to achieve automatic and accurate audiovisual emotion recognition is significantly important for developing engaging and empathetic human–computer interaction environment. However, two major challenges exist in the field of audiovisual emotion recognition: (1) how to effectively capture representations of each single modality and eliminate redundant features and (2) how to efficiently integrate information from these two modalities to generate discriminative representations.

A novel key-frame extraction-based attention fusion network (KE-AFN) is proposed for audiovisual emotion recognition. KE-AFN attempts to integrate key-frame extraction with multimodal interaction and fusion to enhance audiovisual representations and reduce redundant computation, filling the research gaps of existing approaches. Specifically, the local maximum–based content analysis is designed to extract key-frames from videos for the purpose of eliminating data redundancy. Two modules, including “Multi-head Attention-based Intra-modality Interaction Module” and “Multi-head Attention-based Cross-modality Interaction Module”, are proposed to mine and capture intra- and cross-modality interactions for further reducing data redundancy and producing more powerful multimodal representations.

Extensive experiments on two benchmark datasets (i.e. RAVDESS and CMU-MOSEI) demonstrate the effectiveness and rationality of KE-AFN. Specifically, (1) KE-AFN is superior to state-of-the-art baselines for audiovisual emotion recognition. (2) Exploring the supplementary and complementary information of different modalities can provide more emotional clues for better emotion recognition. (3) The proposed key-frame extraction strategy can enhance the performance by more than 2.79 per cent on accuracy. (4) Both exploring intra- and cross-modality interactions and employing attention-based audiovisual fusion can lead to better prediction performance.

The proposed KE-AFN can support the development of engaging and empathetic human–computer interaction environment.

A vast body of literature has documented the negative consequences of stress on employee performance and well-being. These deleterious effects are particularly pronounced for service agents who need to constantly endure and manage customer emotions. The purpose of this paper is to introduce and describe a deep learning model to predict in real-time service agent stress from emotion patterns in voice-to-voice service interactions.

A deep learning model was developed to identify emotion patterns in call center interactions based on 363 recorded service interactions, subdivided in 27,889 manually expert-labeled three-second audio snippets. In a second step, the deep learning model was deployed in a call center for a period of one month to be further trained by the data collected from 40 service agents in another 4,672 service interactions.

The deep learning emotion classifier reached a balanced accuracy of 68% in predicting discrete emotions in service interactions. Integrating this model in a binary classification model, it was able to predict service agent stress with a balanced accuracy of 80%.

Service managers can benefit from employing the deep learning model to continuously and unobtrusively monitor the stress level of their service agents with numerous practical applications, including real-time early warning systems for service agents, customized training and automatically linking stress to customer-related outcomes.

The present study is the first to document an artificial intelligence (AI)-based model that is able to identify emotions in natural (i.e. nonstaged) interactions. It is further a pioneer in developing a smart emotion-based stress measure for service agents. Finally, the study contributes to the literature on the role of emotions in service interactions and employee stress.

The purpose of this paper is to bring together the textual and multimedia opinions, since the use of social data has become the new trend that enables to gather the product reputation traded in social media. Integrating a product reputation process into the companies' strategy will bring several benefits such as helping in decision-making regarding the current and the new generation of the product by understanding the customers’ needs. However, image-centric sentiment analysis has received much less attention than text-based sentiment detection.

In this work, the authors propose a multimedia content-based product reputation framework that helps in detecting opinions from social media. Thus, in this case, the analysis of a certain publication is made by combining their textual and multimedia parts.

To test the effectiveness of the proposed framework, a case study based on YouTube videos has been established, as it brings together the image, the audio and the video processing at the same time.

The key novelty is the implication of multimedia content in addition of the textual one with the goal of gathering opinions about a certain product. The multimedia analysis brings together facial sentiment detection, printed text analysis, opinion detection from speeches and textual opinion analysis.

This study aims to propose the effect of five sub-dimensions of the expected benefits, which include compatibility, social influence, convenience, function and emotion on attitude and behavioral intentions.

A research model including eight hypotheses was tested using 413 samples collected in South Korea.

The data analysis results indicated that the five sub-dimensions of expected benefits aid to enhance attitude, which plays an important role in the formation of behavioral intentions.

This study was designed to empirically identify the important role of expected benefits in the context of drone food delivery services for the first time.

无人机送餐服务的预期效益：对态度和行为意向影响的研究

本研究提出无人机送餐的预期效益, 及其五大维度（兼容性, 社交影响力, 方便, 实用, 以及情感）对消费者态度和行为意向的影响。

本研究样本包括413韩国消费者来检测提出的理论模型以及八项研究假设。

数据分析显示预期效益包括的五大维度可以提高消费者态度。消费者态度在促进行为意向产生了关键性影响。

本论文是有关预期效益在无人机送餐的关键性作用的首次实证研究

The purpose of this paper is to answer the question whether it is possible to recognise the gender of a web browser user on the basis of keystroke dynamics and mouse movements.

An experiment was organised in order to track mouse and keyboard usage using a special web browser plug-in. After collecting the data, a number of parameters describing the users’ keystrokes, mouse movements and clicks were calculated for each data sample. Then several machine learning methods were used to verify the stated research question.

The experiment showed that it is possible to recognise males and females on the basis of behavioural characteristics with an accuracy exceeding 70 per cent. The best results were obtained while using Bayesian networks.

The first limitation of the study was the restricted contextual information, i.e. neither the type of web page browsed nor the user activity was taken into account. Another is the narrow scope of the respondent group. Future work should focus on gathering data from more users covering a wider age range and should consider the context.

Automatic gender recognition could be used in profiling a user to create personalised websites or as an additional feature in automatic identification for security reasons. It might be also considered as a confirmation of declared gender in web-based surveys.

As not all users perceive personalised ads and websites as beneficial, this application requires the analysis of a user perspective to provide value to the consumer without privacy violation.

Behavioural characteristics, such as mouse movements and keystroke dynamics, have already been used for user authentication and emotion recognition, but applying these data to gender recognition is an original idea.

Nowadays, the speech emotion recognition (SER) model has enhanced as the main research topic in various fields including human–computer interaction as well as speech processing. Generally, it focuses on utilizing the models of machine learning for predicting the exact emotional status from speech. The advanced SER applications go successful in affective computing and human–computer interaction, which is making as the main component of computer system's next generation. This is because the natural human machine interface could grant the automatic service provisions, which need a better appreciation of user's emotional states.

This paper implements a new SER model that incorporates both gender and emotion recognition. Certain features are extracted and subjected for classification of emotions. For this, this paper uses deep belief network DBN model.

Through the performance analysis, it is observed that the developed method attains high accuracy rate (for best case) when compared to other methods, and it is 1.02% superior to whale optimization algorithm (WOA), 0.32% better from firefly (FF), 23.45% superior to particle swarm optimization (PSO) and 23.41% superior to genetic algorithm (GA). In case of worst scenario, the mean update of particle swarm and whale optimization (MUPW) in terms of accuracy is 15.63, 15.98, 16.06% and 16.03% superior to WOA, FF, PSO and GA, respectively. Under the mean case, the performance of MUPW is high, and it is 16.67, 10.38, 22.30 and 22.47% better from existing methods like WOA, FF, PSO, as well as GA, respectively.

This paper presents a new model for SER that aids both gender and emotion recognition. For the classification purpose, DBN is used and the weight of DBN is used and this is the first work uses MUPW algorithm for finding the optimal weight of DBN model.

Although phone calls are widely used by debt collection services to persuade delinquent customers to repay, few financial services studies have analyzed the unstructured voice and text data to investigate how debt collection call strategies drive customers to repay. Moreover, extant research opens the “black box” mainly through psychological theories without hard behavioral data of customers. The purpose of our study is to address this research gap.

The authors randomly sampled 3,204 debt collection calls from a large consumer finance company in East Asia. To rule out alternative explanations for the findings, such as consumers' previous experience of being persuaded by debt collectors or repeated calls, the authors selected calls made to delinquent customers who had not been delinquent before and were being called by the company for the first time. The authors transformed the unstructured voice and textual data into structured data through automatic speech recognition (ASR), voice mining, natural language processing (NLP) and machine learning analyses.

The findings revealed that (1) both moral appeal (carrot) and social warning (stick) strategies decrease repayment time because they arouse mainly happy emotion and fear emotion, respectively; (2) the legal warning (stick) strategy backfires because of decreasing the happy emotion and triggering the anger emotion, which impedes customers' compliance; and (3) in contrast to traditional wisdom, the combination of carrot and stick fails to decrease the repayment time.

The findings provide a valuable and systematic understanding of the effect of carrot strategies, stick strategies and the combinations of them on repayment time. This study is among the first to empirically analyze the effectiveness of carrot strategies, stick strategies and their joint strategies on repayment time through unstructured vocal and textual data analysis. What's more, the previous studies open the “black box” through psychological mechanism. The authors firstly elucidate a behavioral mechanism for why consumers behave differently under varying debt collection strategies by utilizing ASR, NLP and vocal emotion analyses.

This study aims to investigate how message sidedness affects the impact of fake news posted on social media on consumers' emotional responses.

The study involves a face-tracking experiment in which 198 participants were exposed to different fake news messages concerning the COVID-19 vaccine. Specifically, participants were exposed to fake news using (1) a one-sided negative fake news message in which the message was entirely unfavorable and (2) a two-sided fake news message in which the negative message was mixed with favorable information. Noldus FaceReader 7, an automatic facial expression recognition system, was used to recognize participants' emotions as they read fake news. The authors sampled 17,450 observations of participants' emotional responses.

The results provide evidence of the significant influence of message sidedness on consumers' emotional valence and arousal. Specifically, two-sided fake news positively influences emotional valence, while one-sided fake news positively influences emotional arousal.

The current study demonstrates that research on fake news posted on social media may particularly benefit from insights regarding the potential but often overlooked importance of strategic design choices in fake news messages and their impact on consumers' emotional responses.