Search results

Children’s emerging conceptions about literacy and its functions are influenced by their experiences with a wide range of written and oral literacies, including the use of digital technology, in their homes and communities. Now that mobile technologies have become intuitive to use, relatively inexpensive, small and easy to move around and networked, they have provided an entry point for transformations in the creation and sharing of texts – they are changing the way young children ‘do’ literacy. In this chapter, the authors discuss the ways that children learn about multimodal texts; how mobile technology can facilitate the reading, creation and sharing of multimodal texts in preschool and primary classrooms; the literacy skills necessary for reading multimodal texts, and; strategies for planning instruction into which multimodal texts and mobile devices are integrated. Examples of how children may engage in multimodal reading and writing in and out of the classroom are also provided.

Purpose – To help teacher educators understand how to more fully prepare pre-service teachers (PSTs) for meaningful and effective instruction with multimodal texts and the underlying technologies.

Design – This mixed methods investigation employed designed-based research in that as the authors observed and gathered data on PSTs’ outcomes within the context of a literacy methods course, the authors also engaged in an iterative process of collaborative design to develop a sustainable instructional model across three academic semesters with three cohorts of PSTs. The authors analyzed pre- and post-PST surveys measuring their knowledge of, disposition toward, and self-efficacy with technology and technology in teaching as well their intent to use technology in their future teaching. The authors also coded and analyzed PST lesson plans completed across each semester for instances of meaningful integration of multimodal texts and the underlying technology, and sound literacy instruction. Finally, the authors closely examined differences in how the course was shaped and “reshaped” across all three iterations and noted any differences in PST outcomes related to these shifts.

Findings – Overall findings suggest that enrollment in the literacy methods course improved both PSTs’ self-efficacy and knowledge about teaching with technology while also supporting PSTs’ ability to develop sound literacy instructional plans. Moreover, strategic positioning of multimodal texts and technology, in which integration is seamless, can help PSTs meaningfully and effectively weave multimodal text sets into their literacy lesson plans.

Practical Implications – This chapter contributes to the literature on integrating multimodal texts and the underlying technologies into PST programs by providing explicit, research-based recommendations for how teacher educators can meaningfully and seamlessly infuse multimodal text sets into core curricula and instructional practices.

In this editorial for a double volume on “Multimodality, Meaning, and Institutions” in Research in the Sociology of Organizations, we aim to achieve three objectives: first, we provide a set of guiding ideas about what a multimodal prism entails for the study of meaning and institutions; second, we give an overview of the topics, concepts, and methods covered in this volume and briefly introduce the central contributions and insights of each article; third, we outline a number of open questions and fruitful avenues for a future research agenda at the intersection of organization studies, institutional theory, and multimodality research.

This research explores two interconnected questions: (1) How do we approach stylistic features of multimodal rhetorical artifacts such as protest posters? (2) Do said artifacts designed for different purposes exhibit systematic stylistic differences? Drawing on Charles Sanders Peirce’s semiotic categorization, this study develops a framework for examining concision, one of the primary stylistic considerations for multimodal rhetorical artifacts such as protest posters. This paper illustrates the use of this framework by exploring the correlation between rhetorical purpose and concision in posters created and disseminated before and during the 2011–2012 Québécois student movement. This study fine-tunes our existing knowledge on multimodality with style sensitivity, and demonstrates how an economy-of-sign based semiotic approach could enrich the empirical examination of multimodal rhetorical artifacts by generating more controlled interpretations.

The digitalization of archival management has rapidly developed with the maturation of digital technology. With data's exponential growth, archival resources have transitioned from single modalities, such as text, images, audio and video, to integrated multimodal forms. This paper identifies key trends, gaps and areas of focus in the field. Furthermore, it proposes a theoretical organizational framework based on deep learning to address the challenges of managing archives in the era of big data.

Via a comprehensive systematic literature review, the authors investigate the field of multimodal archive resource organization and the application of deep learning techniques in archive organization. A systematic search and filtering process is conducted to identify relevant articles, which are then summarized, discussed and analyzed to provide a comprehensive understanding of existing literature.

The authors' findings reveal that most research on multimodal archive resources predominantly focuses on aspects related to storage, management and retrieval. Furthermore, the utilization of deep learning techniques in image archive retrieval is increasing, highlighting their potential for enhancing image archive organization practices; however, practical research and implementation remain scarce. The review also underscores gaps in the literature, emphasizing the need for more practical case studies and the application of theoretical concepts in real-world scenarios. In response to these insights, the authors' study proposes an innovative deep learning-based organizational framework. This proposed framework is designed to navigate the complexities inherent in managing multimodal archive resources, representing a significant stride toward more efficient and effective archival practices.

This study comprehensively reviews the existing literature on multimodal archive resources organization. Additionally, a theoretical organizational framework based on deep learning is proposed, offering a novel perspective and solution for further advancements in the field. These insights contribute theoretically and practically, providing valuable knowledge for researchers, practitioners and archivists involved in organizing multimodal archive resources.

This paper presents a survey of research into interactive robotic systems for the purpose of identifying the state of the art capabilities as well as the extant gaps in this emerging field. Communication is multimodal. Multimodality is a representation of many modes chosen from rhetorical aspects for its communication potentials. The author seeks to define the available automation capabilities in communication using multimodalities that will support a proposed Interactive Robot System (IRS) as an AI mounted robotic platform to advance the speed and quality of military operational and tactical decision making.

This review will begin by presenting key developments in the robotic interaction field with the objective of identifying essential technological developments that set conditions for robotic platforms to function autonomously. After surveying the key aspects in Human Robot Interaction (HRI), Unmanned Autonomous System (UAS), visualization, Virtual Environment (VE) and prediction, the paper then proceeds to describe the gaps in the application areas that will require extension and integration to enable the prototyping of the IRS. A brief examination of other work in HRI-related fields concludes with a recapitulation of the IRS challenge that will set conditions for future success.

Using insights from a balanced cross section of sources from the government, academic, and commercial entities that contribute to HRI a multimodal IRS in military communication is introduced. Multimodal IRS (MIRS) in military communication has yet to be deployed.

Multimodal robotic interface for the MIRS is an interdisciplinary endeavour. This is not realistic that one can comprehend all expert and related knowledge and skills to design and develop such multimodal interactive robotic interface. In this brief preliminary survey, the author has discussed extant AI, robotics, NLP, CV, VDM, and VE applications that is directly related to multimodal interaction. Each mode of this multimodal communication is an active research area. Multimodal human/military robot communication is the ultimate goal of this research.

A multimodal autonomous robot in military communication using speech, images, gestures, VST and VE has yet to be deployed. Autonomous multimodal communication is expected to open wider possibilities for all armed forces. Given the density of the land domain, the army is in a position to exploit the opportunities for human–machine teaming (HMT) exposure. Naval and air forces will adopt platform specific suites for specially selected operators to integrate with and leverage this emerging technology. The possession of a flexible communications means that readily adapts to virtual training will enhance planning and mission rehearsals tremendously.

Interaction, perception, cognition and visualization based multimodal communication system is yet missing. Options to communicate, express and convey information in HMT setting with multiple options, suggestions and recommendations will certainly enhance military communication, strength, engagement, security, cognition, perception as well as the ability to act confidently for a successful mission.

The objective is to develop a multimodal autonomous interactive robot for military communications. This survey reports the state of the art, what exists and what is missing, what can be done and possibilities of extension that support the military in maintaining effective communication using multimodalities. There are some separate ongoing progresses, such as in machine-enabled speech, image recognition, tracking, visualizations for situational awareness, and virtual environments. At this time, there is no integrated approach for multimodal human robot interaction that proposes a flexible and agile communication. The report briefly introduces the research proposal about multimodal interactive robot in military communication.

The first purpose is to assess the quality of containerized multimodal export and the second is to develop and demonstrate the design of a service network with quality approach.

The article used the structural equation model to develop a model to measure the quality of multimodal transportation for containerized exports and finalized the model with an alternative approach. The evolutionary algorithm had been used to design a service network based on quality.

Provided factors affecting quality of multimodal transportation and reverse to one hypothesis, the construct variation in cost, time shape and quantity did not affect the quality of multimodal transportation for containerized exports. The model without variation construct was finalized by exploring causality.

This research had scope till container loading onto the vessel and assessed the quality for containerized cargo only, and second research purpose is limited by assumed values of fitness function and the limited number of nodes, in service network design demonstration.

This research provided a tool to measure the quality of multimodal transportation for containerized exports and demonstrated the field application of the model developed in service network design. This approach included all factors applicable across the container movement. The integrated approach of the article provided an organized method to design a service network for containerized exports.

This work provided the tool to assess the quality of multimodal transportation for containerized exports and developed an approach to design a service network of multimodal transportation based on quality. This approach has considered the factors of multimodal transportation comprehensively in contrast to the optimization approaches based on operation research techniques.

Multimodal writing portfolios were introduced and integrated into an undergraduate course and a graduate course in a research-oriented university in northwest Taiwan. This study aims to examine the influence of multimodal writing portfolios of novice researchers' academic writing.

Comparative case studies involve collecting data from several cases and analyzing the similarities, differences and patterns across cases (Merriam, 2009). To address this underdeveloped area of research, a comparative case study method was employed to understand undergraduate and graduate students' multimodal writing portfolios in academic writing in two courses in Taiwan.

First, multimodal writing portfolios enabled novice researchers to be more familiar with the structure of academic paper and they had better performance in intrapersonal and linguistic aspects. Second, novice researchers held positive attitude toward multimodal writing portfolios because they regarded process of making multimodal writing portfolios as preparation for their future academic writing. Finally, participants highly valued the class PowerPoint slides, weekly writing tasks and the instructor's modeling as effective facilitation for making multimodal writing portfolios.

Limited studies focus on multimodal writing portfolios (e.g. Silver, 2019). The present case study explores the integration of a multimodal writing portfolio into one undergraduate and one graduate course to explore learners' attitude and performance in academic writing.

Novice researchers can learn to compose multimodal academic texts for the academic writing community.

Suggestions on effective integration of multimodal writing portfolios into academic writing instruction were provided based on the research findings.

The findings of the study provide the field of L2 writing with insights into the pedagogical development of multilingual writing portfolios and help educators to be better prepared for teaching novice researchers to comprehend and compose multimodal texts and enter the academic writing community. The framework in Figure 1 and suggestions on course designs for academic writing can inform educators on the integration of multimodality in academic discourse. Moreover, this study moves beyond general writing courses at the tertiary level and could contribute to L2 writers' deeper understanding of how multimodal writing portfolios can be constructed.

Literacy research exploring multimodal composition and justice-oriented children’s literature each have rich landscapes and histories. This paper aims to add to both of these bodies of scholarship through the emerging assemblage of Studio F, a fifth-grade classroom. The authors share poststructural analytic encounters with attention to the unexpected multimodal relationships and the justice-oriented talk and texts that emerged, as well as the classroom conditions that produce them.

The authors think with assemblage theory to examine the newness that emerged as one small group of students wrestled with the emerging instances of racism present in Freedom Summer by Deborah Wiles.

Together, the living arrangement of bodies, materials and discourses created openings for students’ explorations of race and racism.

This paper offers teachers and researchers space to rethink what is possible in the literacy classroom when the authors re-envision classrooms as vibrant assemblages, support emergent multimodal composing processes and follow students’ critical encounters toward justice-oriented literacies.

This paper aims to propose a critical multimodal framework to understanding pedagogical materials that focuses on not only the verbal or the visual components but also the interaction between the two semiotic resources that constructs power relations as a result of intermodal interaction, and it further provides an example of an in-depth analysis of one text using this approach.

The paper proposes a critical multimodal framework that draws from Serafini (2010) and Royce (1998). Details about how the two works complement to form a critical multimodal framework are discussed, after which the paper analyzes one example from an English-language textbook using the proposed framework to demonstrate its strengths.

The findings highlight the power relations constructed in texts as a result of the interaction between the verbal and visual components, specifically how the visual mode functions to rationalize the power relations constructed in the verbal mode. These findings also establish the significance for considering the larger context of materials production and reception identified in the ideological perspective to appreciate how texts reflect discourses in diverse locales.

This paper argues that even though critical multimodality has often been discussed conceptually in L1 literacy scholarship, how to put these conceptualizations into practice has not been addressed systematically. The paper also contends that critical perspectives to understanding multimodal texts are also important in L2 English-language teaching. The critical multimodal framework proposed thus serves as a conceptual and methodological framework for multimodal reading and interpretive practices in both L1 and L2 contexts.