Search results

How can we take multimodalities (the discursive, material, spatial, visual, emotional, embodied, etc.) of institutions seriously? In contemplating the implications of the “multimodal turn” (broadly defined) for institutional inquiry and theory, I first situate it within its intellectual current in the social sciences more broadly. I then use three ethnographic vignettes from Israeli high-tech conferences, all centering on “place” (as a – presumably first and foremost – geographical and material reality) to highlight the shortcomings of a “weak” multimodal approach and the promise of a “strong” one. Finally, I suggest ways to capture multiple modalities within an integrated account and discuss the challenges entailed in an institutional inquiry undertaken to acknowledge, and conceptualize, non-linguistic realities.

In this editorial for a double volume on “Multimodality, Meaning, and Institutions” in Research in the Sociology of Organizations, we aim to achieve three objectives: first, we provide a set of guiding ideas about what a multimodal prism entails for the study of meaning and institutions; second, we give an overview of the topics, concepts, and methods covered in this volume and briefly introduce the central contributions and insights of each article; third, we outline a number of open questions and fruitful avenues for a future research agenda at the intersection of organization studies, institutional theory, and multimodality research.

In this article, we develop and advance an understanding of institutions as multimodal accomplishments. We draw on social semiotics and the linguistic concept of metafunctions to establish the visual as a specific mode of meaning construction. In addition, we make semiotic modes conducive to institutional inquiry by introducing the notion of distinct “modal registers” – specialized configurations of linguistic signs within a particular mode that are adapted and applied in the reproduction of institutions or institutional domains. At the core of our article, we operationalize metafunctions to develop methodology for the analysis of visual registers. We illustrate our approach with data from Corporate Social Responsibility (CSR) reporting in Austria.

With the rapid development of short videos in China, the public has become accustomed to using short videos to express their opinions. This paper aims to solve problems such as how to represent the features of different modalities and achieve effective cross-modal feature fusion when analyzing the multi-modal sentiment of Chinese short videos (CSVs).

This paper aims to propose a sentiment analysis model MSCNN-CPL-CAFF using multi-scale convolutional neural network and cross attention fusion mechanism to analyze the CSVs. The audio-visual and textual data of CSVs themed on “COVID-19, catering industry” are collected from CSV platform Douyin first, and then a comparative analysis is conducted with advanced baseline models.

The sample number of the weak negative and neutral sentiment is the largest, and the sample number of the positive and weak positive sentiment is relatively small, accounting for only about 11% of the total samples. The MSCNN-CPL-CAFF model has achieved the Acc-2, Acc-3 and F1 score of 85.01%, 74.16 and 84.84%, respectively, which outperforms the highest value of baseline methods in accuracy and achieves competitive computation speed.

This research offers some implications regarding the impact of COVID-19 on catering industry in China by focusing on multi-modal sentiment of CSVs. The methodology can be utilized to analyze the opinions of the general public on social media platform and to categorize them accordingly.

This paper presents a novel deep-learning multimodal sentiment analysis model, which provides a new perspective for public opinion research on the short video platform.

The purpose of this study is to examine five Secondary English Language teachers’ perceptions and practices of multiliteracies teaching in the context of a decade after multiliteracies was introduced into the English Language syllabus in Singapore.

Adopting a case study approach, the authors observed 12 multiliteracies lessons taught by the five teacher participants across three secondary schools. The classroom data included field notes and video-recordings of the lessons. The authors also conducted pre-lesson and post-lesson interviews with the teachers to understand their beliefs and the rationale behind their classroom practices. The video-recordings of the lessons and audio-recordings of the interviews were transcribed verbatim and analysed using thematic coding.

The authors identify an essentialising of multiliteracies to the skills of viewing and representing with multimodal texts, as well as a sense of uncertainty amongst the teachers towards the teaching of multiliteracies. In terms of practices, the authors observed an attempt to connect with the students’ life-worlds through the use of authentic materials, but often only in service of language learning. The authors also highlight the constraining influence of assessment on shaping multiliteracies learning. The findings of this study resonate with the conclusions that some of the earlier studies reported on teachers’ perceptions and practices of multiliteracies teaching. This resonation suggests perennial issues and challenges which remain unresolved.

This study is limited by the number of participants as well as the selected lessons the authors observed. The schools, while selected because they were considered as mid-range public schools in Singapore, were also not representative of all Singapore schools. As such, the authors acknowledge that the generalisability of the findings from this study is limited.

The issues raised in this study resonate with the findings from previous studies both from Singapore and around the world. The persistence of these concerns over time and space that remain unresolved demands attention and concerted action from policymakers, curriculum developers and education researchers, to address the challenges in multiliteracies teaching and learning.

This study was conducted a decade after the launch of the English Language Syllabus 2010, which first incorporated multiliteracies into the curriculum. This study examines the teachers’ perceptions and practices in relation to the policy intent. The implications from this study are relevant to educators interested in integrating multiliteracies in the literacy curriculum internationally.

The digitalization of archival management has rapidly developed with the maturation of digital technology. With data's exponential growth, archival resources have transitioned from single modalities, such as text, images, audio and video, to integrated multimodal forms. This paper identifies key trends, gaps and areas of focus in the field. Furthermore, it proposes a theoretical organizational framework based on deep learning to address the challenges of managing archives in the era of big data.

Via a comprehensive systematic literature review, the authors investigate the field of multimodal archive resource organization and the application of deep learning techniques in archive organization. A systematic search and filtering process is conducted to identify relevant articles, which are then summarized, discussed and analyzed to provide a comprehensive understanding of existing literature.

The authors' findings reveal that most research on multimodal archive resources predominantly focuses on aspects related to storage, management and retrieval. Furthermore, the utilization of deep learning techniques in image archive retrieval is increasing, highlighting their potential for enhancing image archive organization practices; however, practical research and implementation remain scarce. The review also underscores gaps in the literature, emphasizing the need for more practical case studies and the application of theoretical concepts in real-world scenarios. In response to these insights, the authors' study proposes an innovative deep learning-based organizational framework. This proposed framework is designed to navigate the complexities inherent in managing multimodal archive resources, representing a significant stride toward more efficient and effective archival practices.

This study comprehensively reviews the existing literature on multimodal archive resources organization. Additionally, a theoretical organizational framework based on deep learning is proposed, offering a novel perspective and solution for further advancements in the field. These insights contribute theoretically and practically, providing valuable knowledge for researchers, practitioners and archivists involved in organizing multimodal archive resources.

The purpose of this study was to explore the viability of transinformation analysis as a multimodal readability metric. A novel approach was called for, considering that existing and established readability metrics are strictly used to measure linguistic complexity. Yet, the corpus of multimodal literature continues to grow, along with the need to understand how non-linguistic modalities contribute to the complexity of the reading experience.

In this exploratory study, think aloud screen recordings of eighth-grade readers of the born-digital novel Inanimate Alice were analyzed for complexity, along with transcripts of post-oral retellings. Pixel-level entropy analysis served as both an objective measure of the document and a subjective measure of the amount of reader information attention. Post-oral retelling entropy was calculated at the unit level of the word, serving as an indication of complexity in recall.

Findings confirmed that transinformation analysis is a viable multimodal readability metric. Inanimate Alice is an objectively complex document, creating a subjectively complex reading experience for the participants. Readers largely attended to the linguistic mode of the story, effectively reducing the amount of information they processed. This was also evident in the brevity and below average complexity of their post-oral retellings, which relied on recall of the linguistic mode. There were no significant group differences among the readers.

This is the first study that uses entropy to analyze multimodal readability.

The purpose of this paper is to reflect on pedagogical strategies which support the teaching of critical analysis of visual and multimodal texts in a tertiary-level course for Arts students.

The paper describes strategies which focus on developing students’ abilities to express interpretive critique, as opposed to mere description. These strategies give students strong scaffolding towards success in their interpretive writing. The course in question is a tertiary-level Arts course which teaches Kress and van Leeuwen’s (2006) approach to “reading images” in relation to contemporary media texts. The basic structure of the course is described, along with the macro steps which underpin the pedagogy. Examples of highly successful and less successful student writing are compared to reveal the key components of effective interpretive answers.

In addition to the normal expectations regarding essay structure and style, and in addition to mastery of the technicality of the course, successful and less successful student writing depends on their mastery of a specific set of moves within the essay. These moves integrate textual observations with clear explanations and a strong relation to interpretation.

While the course and strategies discussed are for tertiary-level students, the strategies described are adaptable to primary and secondary levels also. Multimodal texts are an integral part of the English curriculum, and all teachers need to explore strategies for enabling their students’ critical engagement with such texts.

Visual and multimodal texts are an exciting and also challenging part of English curricula, and new analytical frameworks and pedagogical strategies are needed to tackle these texts. In particular, the gap between simply describing visual resources (applying the tools) and critical analysis (using the tools) is vast, and specific pedagogical strategies are needed to help students develop the necessary interpretive language.