A comparative study of the effectiveness of sentiment tools and human coding in sarcasm detection

Abstract

Purpose

Sarcasm is often used in everyday speech and writing and is prevalent in online contexts. The purpose of this paper is to investigate the analogy between sarcasm comments from sentiment tools and the human coder.

Design/methodology/approach

Using the Verbal Irony Procedure, eight human coders were engaged to analyse comments collected from an online commercial page, and a dissimilarity analysis was conducted with sentiment tools. Three constants were tested, namely, polarity from sentiment tools, polarity rating by human coders; and sarcasm-level ratings by human coders.

Findings

Results found an inconsistent ratio between these three constants. Sentiment tools used did not have the capability or reliability to detect the subtle, contextualized meanings of sarcasm statements that human coders could detect. Further research is required to refine the sentiment tools to enhance their sensitivity and capability.

Practical implications

With these findings, it is recommended that further research and commercialization efforts be directed at improving current sentiment tools – for example, to incorporate sophisticated human sarcasm texts in their analytical systems. Sarcasm exists frequently in media, politics and human forms of communications in society. Therefore, more highly sophisticated sentiment tools with the abilities to detect human sarcasm would be vital in research and industry.

Social implications

The findings suggest that presently, of the sentiment tools investigated, most are still unable to pick up subtle contexts within the text which can reverse or change the message that the writer intends to send to his/her receiver. Hence, the use of the relevant hashtags (e.g. #sarcasm; #irony) are of fundamental importance in detection tools. This would aid the evaluation of product reviews online for commercial usage.

Originality/value

The value of this study lies in its original, empirical findings on the inconsistencies between sentiment tools and human coders in sarcasm detection. The current study proves these inconsistencies are detected between human and sentiment tools in social media texts and points to the inadequacies of current sentiment tools. With these findings, it is recommended that further research and commercialization efforts be directed at improving current sentiment tools – to incorporate sophisticated human sarcasm texts in their analytical systems. The system can then be used as a reference for psychologists, media analysts, researchers and speech writers to detect cues in the inconsistencies in behaviour and language.

Citation

Teh, P.L., Ooi, P.B., Chan, N.N. and Chuah, Y.K. (2018), "A comparative study of the effectiveness of sentiment tools and human coding in sarcasm detection", Journal of Systems and Information Technology, Vol. 20 No. 3, pp. 358-374. https://doi.org/10.1108/JSIT-12-2017-0120