Speaking with a “forked tongue” – misalignment between user ratings and textual emotions in LLMs

Abstract

Purpose

The study aims to provide concrete service remediation and enhancement for LLM developers such as getting user forgiveness and breaking through perceived bottlenecks. It also aims to improve the efficiency of app users' usage decisions.

Design/methodology/approach

This paper takes the user reviews of the app stores in 21 countries and 10 languages as the research data, extracts the potential factors by LDA model, exploratively takes the misalignment between user ratings and textual emotions as user forgiveness and perceived bottleneck and uses the Word2vec-SVM model to analyze the sentiment. Finally, attributions are made based on empathy.

Findings

The results show that AI-based LLMs are more likely to cause bias in user ratings and textual content than regular APPs. Functional and economic remedies are effective in awakening empathy and forgiveness, while empathic remedies are effective in reducing perceived bottlenecks. Interestingly, empathetic users are “pickier”. Further social network analysis reveals that problem solving timeliness, software flexibility, model updating and special data (voice and image) analysis capabilities are beneficial in breaking perceived bottlenecks. Besides, heterogeneity analysis show that eastern users are more sensitive to the price factor and are more likely to generate forgiveness through economic remedy, and there is a dual interaction between basic attributes and extra boosts in the East and West.

Originality/value

The “gap” between negative (positive) user reviews and ratings, that is consumer forgiveness and perceived bottlenecks, is identified in unstructured text; the study finds that empathy helps to awaken user forgiveness and understanding, while it is limited to bottleneck breakthroughs; the dataset includes a wide range of countries and regions, findings are tested in a cross-language and cross-cultural perspective, which makes the study more robust, and the heterogeneity of users' cultural backgrounds is also analyzed.

Citation

Yang, Y. and Huang, J. (2024), "Speaking with a “forked tongue” – misalignment between user ratings and textual emotions in LLMs", Kybernetes, Vol. ahead-of-print No. ahead-of-print. https://doi.org/10.1108/K-06-2024-1458