To read this content please select one of the options below:

Exploring the influence of multimodal social media data on stock performance: an empirical perspective and analysis

Hui Yuan (Shanghai International Studies University, Shanghai, China)
Yuanyuan Tang (Renmin University of China, Beijing, China)
Wei Xu (Renmin University of China, Beijing, China)
Raymond Yiu Keung Lau (City University of Hong Kong, Kowloon Tong, Hong Kong)

Internet Research

ISSN: 1066-2243

Article publication date: 12 January 2021

Issue publication date: 19 May 2021




Despite the extensive academic interest in social media sentiment for financial fields, multimodal data in the stock market has been neglected. The purpose of this paper is to explore the influence of multimodal social media data on stock performance, and investigate the underlying mechanism of two forms of social media data, i.e. text and pictures.


This research employs panel vector autoregressive models to quantify the effect of the sentiment derived from two modalities in social media, i.e. text information and picture information. Through the models, the authors examine the short-term and long-term associations between social media sentiment and stock performance, measured by three metrics. Specifically, the authors design an enhanced sentiment analysis method, integrating random walk and word embeddings through Global Vectors for Word Representation (GloVe), to construct a domain-specific lexicon and apply it to textual sentiment analysis. Secondly, the authors exploit a deep learning framework based on convolutional neural networks to analyze the sentiment in picture data.


The empirical results derived from vector autoregressive models reveal that both measures of the sentiment extracted from textual information and pictorial information in social media are significant leading indicators of stock performance. Moreover, pictorial information and textual information have similar relationships with stock performance.


To the best of the authors’ knowledge, this is the first study that incorporates multimodal social media data for sentiment analysis, which is valuable in understanding pictures of social media data. The study offers significant implications for researchers and practitioners. This research informs researchers on the attention of multimodal social media data. The study’s findings provide some managerial recommendations, e.g. watching not only words but also pictures in social media.



The authors appreciate the editors’ and the anonymous reviewers’ valuable comments. This work was supported in part by the National Natural Science Foundation of China (Grant Nos. 72001144, 71771212, and U1711262), and Innovative Research Team of Shanghai International Studies University (No. 2020114044).


Yuan, H., Tang, Y., Xu, W. and Lau, R.Y.K. (2021), "Exploring the influence of multimodal social media data on stock performance: an empirical perspective and analysis", Internet Research, Vol. 31 No. 3, pp. 871-891.



Emerald Publishing Limited

Copyright © 2020, Emerald Publishing Limited

Related articles