Search results

This paper aims to explore recent research (2007-2017) on the implementation of multimodal writing instruction in secondary English courses. It seeks to highlight the varied ways in which theoretical conceptions of multimodality have been implemented in writing instruction and the impacts of these implementations on students' experiences in classrooms.

The author used a keyword search of relevant academic databases to identity articles within the search parameters. This was followed by bibliographic branching to identify additional articles and two rounds of open coding to identify themes for analysis.

The literature revealed a diversity of approaches to incorporating multimodal writing in classrooms; teachers mixed modalities within assignments, paired writing in print with multimodal composition and redesigned entire units or courses around multimodality. Studies showed the impact of multimodality on student learning through shifts in conceptions of communication, increases in student engagement, composition for real audiences and an increased role for students’ interests and identities.

This review has implications for teachers and researchers interested in developing multimodal writing curricula. It highlights the specific ways in which multimodal writing can be incorporated into instruction and the changes in student learning that result from this shift.

While theoretical writing on multimodality is abundant, multiple researchers have noted the difficulty of finding research on classroom implementations of multimodality (Howell et al., 2017; Smith, 2017). This review is intended to address this difficulty by contributing to a body of literature that teachers and scholars can draw on as they conceptualize and design multimodal writing experiences for students in the future.

While recent theoretical discussions around discourse–material relationality have facilitated important conceptual and analytical advancements within the broader field of CMS, less progress has been made methodologically with regard to innovating empirical methods and data modes. Therefore, the purpose of this article is to contribute to strengthening the methodological focus in the literature when grappling with the relationality of discourse–materiality and co-constitution. This includes a method-retooling framework inspired by new materialism.

In this article, the authors engage at the methodological level by developing a method-retooling framework that combines insights from organizational discourse studies and new materialist thinking. This framework enables a retooling of existing methods to become sensitive to multimodality and offers two concrete examples that were developed during fieldwork for a multi-sited and multi-method case study in 2018.

Based on the framework for retooling methods for multimodality, two illustrations are offered. These include retooling interviews by employing multimodal vignettes and retooling observations by using multimodal mappings. They are unfolded and discussed regarding their appropriation of discourse–material relationality.

This paper includes original research and method developments – adding a critical focus on the methodological aspects and potential advancements that are necessary in the wake of the ongoing debates around discourse–materiality across CMS and specifically within studies of organizational discourse and CCO. By suggesting a framework, the authors stimulate methodological explorations and contribute to furthering method developments that are equal to the rich conceptual progress made within the field.

This research studies a plastic recycling system from a reverse logistics angle and investigates the potential benefits of a multimodality strategy to the network design of plastic recycling. This research aims to quantify the impact of multimodality on the network, to provide decision support for the design of more sustainable plastic recycling networks in the future.

A MILP model is developed to assess different plastic waste collection, treatment and transportation scenarios. Comprehensive costs of the network are considered, including emission costs. A baseline scenario represents the optimized current situation while other scenarios allow multimodality options (barge and train) to be applied.

Results show that transportation cost contributes to about 7 percent of the total cost and multimodality can bring a reduction of almost 20 percent in transportation costs (CO₂‐eq emissions included). In our illustrative case with two plastic separation methods, the post‐separation channel benefits more from a multimodality strategy than the source‐separation channel. This relates to the locations and availability of intermediate facilities and the quantity of waste transported on each route.

This study applies a reverse logistics network model to design a plastic recycling network with special structures and incorporates a multimodality strategy to improve sustainability. Emission costs (carbon emission equivalents times carbon tax) are added to the total cost of the network to be optimized.

The purpose of this research is to explore how an industry is represented in multimodal public media narratives and to explore how this representation subsequently affects the formation of public sense-giving space during a persisting crisis, such as a pandemic. The question asked is: how do the use of multimodality by public service media dynamically shape representations of industry identity during a persisting crisis?

This study made use of a multimodal approach. The verbal and visual media text on the restaurant industry during the COVID-19 pandemic that were published in Finland by the public service media distributor Yle were studied. Data published between March 2020 and March 2022 were analysed. The data consisted of 236 verbal texts, including 263 visuals.

Three narratives were identified– victim, servant and survivor – that construct power relations and depict the identity of the restaurant industry differently. It was argued that multimodal media narratives hold three meaning making functions: sentimentalizing, juxtaposing and nuancing industry characteristics. It was also argued that multimodal public service media narratives have wider implications in possibly shaping the future attractiveness of the industry and organizational members' understanding of their identity.

This research contributes to sensemaking literature in that it explores the role of power – explicitly or implicitly constructed through media narratives during crisis. Furthermore, this research contributes to sensemaking literature in that it shows how narratives take shape multimodally during a continuous crisis, and how this impacts the construction of industry identity.

Our world had always been multimodal, but studying how young children enact and embody literacy practices, especially reading, has often been overlooked. The purpose of this study was to examine how young children respond to nonfiction picturebooks in multimodal ways. This paper aims to answer the question: What multimodal resources do readers use to respond to and construct meaning from nonfiction picturebooks?

Undergirded by Rosenblatt’s transactional theory of reading and social semiotic multimodality, a 9-min video clip of three boys making sense of one nonfiction picturebook during reading workshop was analyzed using Norris’ approach to multimodal data analysis. This research stemmed from a five-month-long case study of one kindergarten class’s multimodal and collective responses to nonfiction picturebooks.

Findings demonstrate how readers use gesture, gaze and proxemics in addition to language to signal agreement with one another, explain new ideas or concepts to one another and incorporate their background knowledge. In addition to reading images, the children learned to read each other.

This research indicates that reading is inherently multimodal, recursive and complex and provides implications for teachers to reconsider what kinds of responses they prioritize in their classrooms. Additionally, this research establishes the need to better understand how readers respond to nonfiction books and a broader examination of multimodality in the literacy curriculum.

This paper presents a survey of research into interactive robotic systems for the purpose of identifying the state of the art capabilities as well as the extant gaps in this emerging field. Communication is multimodal. Multimodality is a representation of many modes chosen from rhetorical aspects for its communication potentials. The author seeks to define the available automation capabilities in communication using multimodalities that will support a proposed Interactive Robot System (IRS) as an AI mounted robotic platform to advance the speed and quality of military operational and tactical decision making.

This review will begin by presenting key developments in the robotic interaction field with the objective of identifying essential technological developments that set conditions for robotic platforms to function autonomously. After surveying the key aspects in Human Robot Interaction (HRI), Unmanned Autonomous System (UAS), visualization, Virtual Environment (VE) and prediction, the paper then proceeds to describe the gaps in the application areas that will require extension and integration to enable the prototyping of the IRS. A brief examination of other work in HRI-related fields concludes with a recapitulation of the IRS challenge that will set conditions for future success.

Using insights from a balanced cross section of sources from the government, academic, and commercial entities that contribute to HRI a multimodal IRS in military communication is introduced. Multimodal IRS (MIRS) in military communication has yet to be deployed.

Multimodal robotic interface for the MIRS is an interdisciplinary endeavour. This is not realistic that one can comprehend all expert and related knowledge and skills to design and develop such multimodal interactive robotic interface. In this brief preliminary survey, the author has discussed extant AI, robotics, NLP, CV, VDM, and VE applications that is directly related to multimodal interaction. Each mode of this multimodal communication is an active research area. Multimodal human/military robot communication is the ultimate goal of this research.

A multimodal autonomous robot in military communication using speech, images, gestures, VST and VE has yet to be deployed. Autonomous multimodal communication is expected to open wider possibilities for all armed forces. Given the density of the land domain, the army is in a position to exploit the opportunities for human–machine teaming (HMT) exposure. Naval and air forces will adopt platform specific suites for specially selected operators to integrate with and leverage this emerging technology. The possession of a flexible communications means that readily adapts to virtual training will enhance planning and mission rehearsals tremendously.

Interaction, perception, cognition and visualization based multimodal communication system is yet missing. Options to communicate, express and convey information in HMT setting with multiple options, suggestions and recommendations will certainly enhance military communication, strength, engagement, security, cognition, perception as well as the ability to act confidently for a successful mission.

The objective is to develop a multimodal autonomous interactive robot for military communications. This survey reports the state of the art, what exists and what is missing, what can be done and possibilities of extension that support the military in maintaining effective communication using multimodalities. There are some separate ongoing progresses, such as in machine-enabled speech, image recognition, tracking, visualizations for situational awareness, and virtual environments. At this time, there is no integrated approach for multimodal human robot interaction that proposes a flexible and agile communication. The report briefly introduces the research proposal about multimodal interactive robot in military communication.

The article seeks to enrich the body of research on store atmospherics by identifying how storefront window design impacts store entry decisions. An innovative multimodal design approach is presented, considering both visual and verbal constituents.

Study 1 draws on a corpus of high-end storefront windows to create a categorization regarding different levels of verbo-visual complexity. The survey in Study 2 (n = 234) serves two purposes: first, to confirm these levels of complexity and second, to investigate the relation between the complexity of window design and store entry decisions.

Study 2 confirms the order of complexity established in Study 1. The results reveal an inverted-U relationship between window complexity and store entry propensity. Windows of medium level of complexity produce shoppers' relatively highest store entry propensity.

The findings suggest that retailers would benefit from adopting verbo-visual window designs of medium complexity, as this combination optimizes the likeliness of consumers to enter stores.

Research on store atmospherics has until recently primarily focused on in-store cues. Studies on store windows remain vastly underrepresented in extant scholarship. The article not only fills this gap but also incorporates an original interdisciplinary angle on multimodality, which offers new methodological perspectives for research in retail and distribution scholarship.

This paper aims to investigate adolescent English as a foreign language (EFL) learners’ digitally mediated multimodal compositions across different genres of writing.

Three Korean high school students participated in the study and created multiple multimodal texts over the course of one academic semester. These texts and other materials were the basis for this study’s qualitative case studies. Multiple sources of data (e.g. class observations, demographic surveys, interviews, field notes and students’ artifacts) were collected. Drawing upon the inductive approach, a coding-oriented analysis was used for the collected data. In addition, a multimodal text analysis was conducted for the students’ multimodal texts and their storyboards.

The study participants’ perceptions of multimodal composing practices seemed to be positively reshaped as a result of them creating multimodal texts. Some participants created multimodal products in phases (e.g. selecting or changing a topic, constructing a storyboard and editing). Especially, although the students’ creative processes had a similarly fixed and linear flow from print-based writing to other modalities, their creative processes proved to be flexible, recursive and/or circular.

This study contributes to the understanding of adolescent English language learners’ multimodal composing practices in the EFL context, which has been underexplored in the literature. It also presents the students’ perspectives on these practices. In short, it provides theoretical and methodological grounds for future L2 literacy researchers to conduct empirical studies on multimodal composing practices.

This paper aims to examine language learners’ critical multimodal literacy practices with a moving-image text, focusing on text comprehension and interpretation rather than text production. It takes a critical perspective towards multimodality and proposes the simultaneous emphasis on critical and multimodal literacies.

This qualitative teacher-inquiry adopts critical multimodal literacy as the framework for understanding learners’ literacy practices. The course implementation highlights images, sounds and words as encompassing the five modes of visual, aural, linguistic, gestural and spatial (Arola et al., 2014) in emphasizing the multimodal in critical multimodal literacy, and the purposeful organization of the images, sounds and words as reflecting the critical in critical multimodal literacy. The analysis also adopts Serafini’s (2010) concentric perceptual, structural and ideological perspectives as the tenets of critical multimodal literacy.

The findings show that focusing on images, sounds, words and their purposeful organization enabled the students to critically examine a moving-image text through considerations for the multiple modes and arriving at the structural and ideological interpretive perspectives.

This study fills a gap in the literature, as very little research has been done to investigate the ways in which language learners engage with, that is, comprehend and interpret, moving-image multimodal texts. In addition, it presents a critical multimodal literacy framework based on Serafini’s (2010) tripartite perspectives and offers pedagogical suggestions for incorporating critical multimodal literacy in language classrooms.