Understanding technology use through multimodal layers: a research review

Purpose – The purpose of this study is to examine the use of digital technologies by teachers and students in teaching and learning from a multimodal layer perspective. Design/methodology/approach – The article reviews 64 studies on technology use. A content analysis based on the theoretical concepts of “ multimodal layers ” was used to synthesise previous research. Findings – Thefindingsindicatethattheuseoftechnologyinclassroompracticesbyteachersandstudentsis multifaceted and that transitions exist between technologies and sign-systems and are differently related to sign-making activities and thus constitute different uses. Between layers, traces can be made that connect the use of technology to differences in sign-making activities. Practical implications – A multimodal layer perspective on technology use is fruitful to understand what happensattheintersectionoftechnologyandhumanactivitiesinschoolpractices.Moreover,moreattentiontomultimodallayerscaninformfutureeffectivetechnologyusageanddesign. Originality/value – The review offers comprehensive insights on how previous research has studied technology using multimodal layers as an analytical lens.


Introduction
Integration of digital technology into education has become a natural part of the school's everyday work, which calls for a more comprehensive understanding of the relationship between their nature, the use and their users (Bezemer and Kress, 2016). Recent studies have focussed on technologies' visual features related to sign-systems (i.e. semiotic resources/modalities), which link technologies, and technologies and individuals in use (Jewitt, 2017). However, so far, many studies have not been considering various components of the technologies, the combinations of these components in use, and the transitions between them that affect learning in different ways (Ravelli and van Leeuwen, 2018). There is also some confusion in research and school practice in terms of how meaning-making actually happens when using technology in teaching and learning, especially how to trace the connections between technologies, their properties and the outcomes (Kress et al., 2014). Jewitt (2002) acknowledged how technologies and sign-systems interrelate and vary in usage from the notion of transitions between mediums. She found that the visual medium of the hardware screen, combined with software CD-ROM, resulted in the formerly written text being transformed into a visual multimodal representation, traced in individuals' sign-making. Sign-making is an ongoing process that, e.g. occurs when signs are prompted from the external world, interpreted by the individual and transformed either into an inward meaning or outward materialization of new representation (Kress, 2010).
Similarly, Adami (2010) emphasized transitions and traces related to the functional and semiotic properties of smartphones. Sign-systems such as labels, keys in different colours and screens of different sizes were activated in meaning-making by touch and oral representations. More recently, these tenets were brought up by Jewitt et al. (2020), who found that there is a transition between technologies that is needed for shifting from visual to other sensory modalities. Furthermore, Adami (2014) explored transitions and traces in relation to software and found that the functional properties of software, such as anchors, make an interactive site for actions of accessing and transferring texts. Moreover, Vigild Poulsen and Kv ale (2018) studied Instagram and the transitions and traces between the use of functional properties of the icons to highlight parts of pictures with semiotic resources such as colours to convey meaning in visual modes of representation.
The notions of transitions and traces appear central to understand technology use in signmaking activities. To obtain a more comprehensive image on such matters, the purpose of this research review is to map and synthesise how previous studies have investigated the use of technology in school settings from a multimodal layer perspective, with a special focus on the variations and connections in usage between technologies, technologies' properties and human's sign-making. A multimodal layer approach is developed in this paper for that purpose, based on a combination of the social semiotic multimodal approach proposed by, e.g. Bezemer and Kress (2016), Kress (2010) and Wartofsky's (1973) artefact taxonomy, consisting of five main elements: technologies, technologies' functional and semiotic properties, and modes of representations made by individuals in activities. The research question is, From a multimodal layer perspective, how has previous research explored teachers' and students' use of technologies in sign-making activities in K-12 schools?
Theoretical outlinemultimodal layers In this section, the approach of multimodal layers is presented, focussing on the five main elements.

Technologies
The use of technologies is related to artifacts, humans' perception and interpretation of representations. Wartofsky (1973) argued that perception is a highly evolved human trait that not only refers to biological senses and projections on the retina but is also mediated by means of our representation, the artifacts we create, which refer to humans activities to engrave sign-systems in physical and symbolic form. When perceived and interpreted according to our socially and culturally shaped knowledge, they will evoke new representations and actions (Kress, 2010;Wartofsky, 1973). This is relevant to understand digital technologies as artifacts in this way, especially since previous research tends to separate the physical from the symbolic, i.e. software is often seen as not being physically manipulated in use, and hardware as a physical rather than a semiotic artifact. The separation of the semiotic features from the physical hardware have neglected the intimate interrelations between hardware and software technologies without acknowledging that various combinations of hardware and software prompt sign-systems differently and affect what is visualized. This has led to an underestimation of the inherent sign-systems of different artifacts that facilitate human needs through multimodal communication (Ravelli and van Leeuwen, 2018). Stressing the semiotic features of technology use offers an analytical lens to understand how the combined use of hardware and software in relation to signsystems may contribute to shaping meaning (Djonov and van Leeuwen, 2013).

IJILT 37,5
Technologies' functional properties Artifacts have different functions in use (Innis, 2013;Scheele, 2006). Wartofsky (1973) explained the relationships between representations engraved in artifacts and their purposes and functions in usetaxonomized into primary, secondary and tertiary levels, here with focus on sign-systems (Innis, 2013;Vigild Poulsen, 2018). The functions are not fixed but in constant flux transitions are made between the different levels related to the users' activities with different technologies and their combinations that render affordances (Kaptelinin and Nardi, 2009).
Wartofsky's proposal of functional properties is significant to understand humans' meaning-making when using multirepresentational digital technologies. The primary functional property is related to sign-systems that convey modes of practice and are linked to production and use (Cole, 1996;Wartofsky, 1973). This kind of representations evokes actions and work but is not immediately connected to patterns of thinking, and is hard to relate to more abstract forms of meaning-making such as learning (Adami, 2014;Wartofsky, 1973). The secondary function, as it preserves or transmits the mode of action by its sign-based properties, provides models and systems to organize production and work (Wartofsky, 1973), for instance, a template for organizing documentation in a specific way (Alnervik, 2018), or a PowerPoint for arranging text by the use of bullet points (Zhao and van Leeuwen, 2014). The tertiary function is more implicitly related to modes of action (Wartofsky, 1973). Cole (2019) asserted that the tertiary function is separated from its direct representational form and has an abstract meaning. For instance, it can be related to the online activity of using the colours inherent in the function of the technology's toolbar, while attending to the offline intention of conveying abstract meaning through the resources hue or saturation (Kress, 2010).

Technologies' semiotic properties
From the social semiotic perspective, it is important to address the relationships between technologies' functions, the semiotic resources used and humans' representations (Vigild Poulsen, 2018). The semiotic properties are something "in-between" that mediate the use of technologies on the tertiary functional level (Wartofsky, 1973) through making the semiotic resources available for semiotic production that goes beyond the designed functional properties. For example, using the semiotic properties of zooming to negotiate certain content in speech and writing. Semiotic properties are therefore the link that additionally enables observing the traces and transitions between technologies' functions and the outcome in signmaking activities (Kress et al., 2014).

Modes of representation
Sign-systems engraved in physical and symbolic artefacts are the precondition for internal sign-making and development of cognitive processing and learning (Bezemer and Kress, 2016). This is a circular process that can be referred to as the ongoing human activities of internalization and externalization (Vygotskij, 1978) signs are prompted by technologies and humans, internalized into mental processing and externalized into new signs. This process transforms the resources available, and the transformations are viewed as signs of learning (Kress, 2010). Representations made by individuals have traces related to the external sign-systems that were considered visually prominent and of interest in the use of technologies in own sign-making activities (Kress et al., 2014), in connection with the notions of foregrounding/backgrounding of information (van Leeuwen, 2005).

Activities
Activities relate to a context consisting of actors, social action and semiotic work (Kress, 2010). Contexts have a specific set of features and means of relating entities (e.g. technologies) Understanding the use of technology that are distinct to that activity site (Norris and Jones, 2005). Kress and van Leeuwen (2001) related the means to the concept of discourse, which is the knowledge and entities developed in a specific social context that is appropriate for and maintained by the actors in that environment. Activities in learning environments are enclosed by a subject as an example of a discourse (Jewitt, 2009) and instructional or learning design principles (Selander and Kress, 2017).

Method
To gain a comprehensive understanding of technology use in sign-making activities, this paper investigates how previous researchers have studied technology use in schools in relation to multimodal layers. The study design followed recommendations proposed by Gough et al. (2012): establishing the target of the research area and questions, creating a data log, formulating inclusion and exclusion criteria, conducting pilot searches, developing the search strategy and identifying relevant literature. These actions were followed by text searching, which in turn entailed the construction of refined selection criteria, as well as screening, assessment and selection of the appropriate articles. The selected research articles were coded and analysed with the multimodal layer framework.

Search strategies
The articles were retrieved using educationally oriented databases: Scopus, Eric and Academic Search Elite. The first cycle of searches was executed at the end of May 2019 through the Scopus database, rendering 352 articles, and the second cycle was carried out at the beginning of June, covering 622 articles from the Eric and Academic Search Elite databases.
The main search terms used were "digital tools" and "digital resources", as these were regarded as the most frequently and interchangeably addressed in research on digital technologies in education. Because these terms are commonly and widely used and refer to both hardware and software, they enabled collection of a large quantity of potential articles. Moreover, because of the breadth of studies addressing various aspects of "use" of technologies in teaching and learning, a decision was taken to exclude this term in the search string. As a result, the final search string was: ((digital AND tools OR digital AND resource) AND teacher AND student AND NOT "higher education" AND NOT university).

Selection of articles
To conduct a valid selection of articles, and to reach a consensus amongst the researchers, an iterative process was undertaken to refine the selection criteria. Initially, general inclusion and exclusion criteria were defined as connected to the purpose of the systematic review on peer-reviewed empirical articles in English, published from 2010 to 2019.
From the two searches, 974 articles were added to Mendeley, and, as duplicates were found, the articles were reduced to 796. From the general selection criteria and by making judgements on the quality and relevance of the found studies, the process started with assessing the first 100 articles. The three authors independently read the abstracts and followed a two-step process, starting with the first 43 articles: (1) individually labelled them with "include", "exclude" or "maybe" in an Excel document; (2) after discussion, a joint decision on the selection criteria was taken. The remaining 57 articles were independently assessed following the same procedure. This resulted in a joint decision on inclusion and exclusion of the articles displayed in Table 1.
The remaining 696 articles' abstracts were divided equally between the authors for inclusion or exclusion assessment based on the final selection criteria using the program IJILT 37,5 Rayyan. General exclusion categories were adapted during a second round, and articles in languages other than English, duplicate articles and shorter articles from forums, columns and journals that no longer exist were removed. This process resulted in 391 articles. To enhance reliability in the coding process, one of the authors selected five articles as good and less good examples of the selection criteria. After discussing these examples, a delimitation was made on articles according to the relevance to "digital tools" and "digital resources", which resulted in 105 articles. The three authors divided the remaining 105 articles for fulltext reading with the aim to review their correspondence with all the inclusion criteria, which resulted in 64 articles being included in this paper.

Data analysis
A simple version of content analysis (Silverman, 2006) was adopted for data coding in the program NVivo 12. The first step of the coding process used the main categories from the multimodal layers and paired them for analysis (visible in Tables 2-8): technologies (divided into categories of hardware and software), technologies' functional properties (primary, secondary, tertiary), technologies' semiotic properties (modalities/semiotic resources), modes of representation (modalities/semiotic resources) and activities. Combinations of hardware and software are important examples of technologies used in meaning-making, why the initial division into different hardware was the first categorization, under which the other layers were organized. All layers where examined from the principles of foregrounding and backgrounding of information (van Leeuwen, 2005), i.e. information explicitly and more frequently described was deemed foregrounded and coded, while other information less salient, deemed more backgrounded and overlooked.

Results and analysis
This section presents findings from the multimodal layers and the numbers of examples.

Students' use
Rich data on students' use of PCs (see table 2) related to mostly visual and linguistic software, the traces of primary functions of, for example, displaying and recording, to the use of the All school forms between kindergarten and school year 12 The purpose is to investigate technology use in K-12 schools with exclusion of higher education. Other contexts, such as from purely technical aspects, teacher surveys, within healthcare issues or from a context that only reflects use in private life/leisure time Articles Scholarly peer-reviewed, full-text empirical studies and book chapters, qualitative and quantitative Systematic reviews, reviews, shorter articles from forums, columns and magazines Table 1.

Final selection criteria
Understanding the use of technology secondary functions to organize such actions, by editing and cutting videos and sound, and navigating content by using hotkeys was observed. These actions are used with the tertiary function of adding special effects in video making and manipulating visual components using semiotic properties such as colours, background, etcetera to convey specific meaning in mainly visual and linguistic modalities (Wartofsky, 1973). The affordances of the technologies' sign-systemsspatiality and simultaneity (visual), kineikonic with combination of different modes such as sound in recording and sequencing (linguistic)can therefore give clues to individuals' semiotic actions and production in the same modalities (Kress, 2010;Kress et al., 2014).
Unlike PCs, smartphones (see table 3) evoke other kinds of use and actualize transitions between modes of representation and technologies. Incorporated in mostly collaborative activities, visual and unnamed software such as web pages were observed in the use of primary functions, such as navigating and scrolling on the touchscreen to access information. Often, auditory modes of representation were generated in use, indicating that smartphones are not yet fully incorporated in other sign-making activities, which need to be manifested in time and space (e.g. linguistic) (Kress, 2010). Because it is mostly feasible to trace actions but not meanings from primary functions (Wartofsky, 1973), smartphones are considered in volatile involvement in speech, prior to engaging in other activities, not least shown in the absence of tertiary functional properties. Individual work: managing tasks and assignments separately Note(s): a Refers to number of mentioned examples in the articles (i.e. not number of articles) b Software where more than one modality is foregrounded in use c Humans' representations where people clearly place emphasis on the use of more than one modality Table 2. Students' use of PCs IJILT 37,5 In students' collaborative use of IWBs (see table 4), yet other traces could be observed, and transitions were visible between properties and modes of representation. Unnamed software was used with the primary capabilities of the interactive board to write and draw, organized through the secondary function of hiding and revealing content. The tertiary function was used to layer typographic semiotic properties to convey meaning, such as that of graphic structure and volatile auditory modalities including speech and gestures, into multimodal compositions (Kress, 2010;Wartofsky, 1973).
The transition and traces found in the use of tablets (see table 5) are similar to those found for PCs. A combination of visual and linguistic software was used, mainly for their primary functions such as displaying, and recording, followed by the secondary function of organizing modes of action such as taking photos, inserting and zooming. However, when the  Understanding the use of technology visual, linguistic and auditory modes of representation come about in combined activities in which the recording function is used as a tertiary function in writing activities, the semiotic properties are used to alter fonts in text or engage with typographic elements to combine with the recordings into multimodal ensembles (Kress, 2010;Wartofsky, 1973).

Teachers' use
Teachers' use of PCs (see table 6) is mainly in instructional design, using their primary function to display and record to create and disseminate content to students (Selander and Kress, 2017). Because it is mostly feasible to trace actions but not meaning from primary functions (Adami, 2014), connections between software technologies and the modes of representation in the same modalities might provide insights into meaning created.
Teachers' use of IWBs (see table 7) is similar to students' use and demonstrated by unifying the actors in learning design activities (Selander and Kress, 2017). The chain of layers enables traces from visual software used with the primary displaying function to organizing the visual properties by the secondary function to, for example, hide and reveal and illuminate important parts. By the tertiary spotlight function, combinations were made with certain semiotic properties, such as colours, to signify visual meaning that was collaboratively negotiated in volatile auditory modes of representation such as speech (Kress, 2010;Wartofsky, 1973).
Teachers' use of tablets (see table 8) is limited in previous research. Undertaken in mainly instructional design, auditory modes of representation complemented by gaze and gestures were used in engagement with the primary and secondary functions to display content to students and organize further learning actions by using, for example, the zoom function. When using the tertiary functions of making recordings in the design of learning material, multimodal affordances of the technologies were combined with similar modes of representation into multimodal compositions (Kress, 2010;Wartofsky, 1973 Multiplicity of modes: combination of auditory-linguisticvisual-gaze-touch Gaze: 2 Gaze: directing attention Touch: 2 Touch: touching the screen Activities STEM: 5 STEM: science Collaboration: 4 Collaboration: shared production Literacy: 3 Literacy: creating e-books Individual: 3 Individual: managing tasks and assignments separately  Understanding the use of technology incorporated into the same activities, which is why differences or similarities between activities and the other layers are difficult to distinguish, important to investigate further. In line with Jewitt (2009) on the visual affordances of technologies, the findings indicate a strong focus on visual, closely accompanied by linguistic modes of representation. An overall tendency is oriented towards multimodality. Gaze and gestures, and sensory modality of touch is depicted in relation to students' use of IWBs, tablets and smartphones combined with visual, linguistic and multimodal software (Jewitt et al., 2020).
Below, the notions of transitions and traces in relation to the multimodal layers will be stressed in discussing the findings.

Traces
According to Kress et al. (2014), if influencing factors are defined, proposed here by the framework of multimodal layers, traces between the technologies and their properties and humans' modes of representation and activities can be illuminated to understand the outcome of use. Examples of traces can be found when the links between the layers are intact. For example, when a hardware is combined with a specific software, the primary functions used in work, and secondary functions used to organize the work, provide possibilities for the tertiary functions to add meaning to the actions. Links can be made between the sign-systems used in the functional properties and human interpretation of them; actions that will be additionally enhanced by the connection to semiotic properties. The choices made amongst all sign-systems can be related to the making of abstract meaning (Cole, 2019) and are relative to the affordances of the properties and their meaning potentials and how they are ultimately drawn upon. Yet, the traces give clues to which of all the multiple features are regarded as visually salient, choices that are integrated into mental processing and renewed and communicated outwardly in sign-making activities (Kress, 2010;Kress et al., 2014). The circular process of internalization and externalization of interpreting and transforming signsystems constitutes learning in connection with technology use (Bezemer and Kress, 2016;Vygotskij, 1978). In this study, these kinds of traces are observed. For instance, in students' use of PCs and tablets, properties are demonstrated to come about differently in the  (Kress, 2010;Kress et al., 2014). Second, the choices made by the individual incorporating several traits from the original input will be a new form meaning made. Hence, although it is not possible here to conduct a detailed analysis, it is important to understand how functional properties are used for action and how they merge with the semiotic properties, as well as how the signsystems therefore are selected and used to highlight specific aspects of meaningin particular, if certain functional properties can be paired with specific semiotic properties, and how they relate to meaning created in a specific context. When one layer is left out, the links between the layers are broken and makes traces harder to observe. It is clear that if only primary functional properties are used, the semiotic properties are left out, and difficulties in relating technology use to sign-making activities arise (Adami, 2014). Hence, the online actions associated with the use of properties cannot be related to the offline activities of meaning-making (Wartofsky, 1973). In summary, traces made between all layers, can give comprehensive clues to how technologies relate to the construction of meaning through the sign-systems, a recognition important to acknowledge in learning settings.

Transitions
The notion of transitions refers to the process of negotiation and selection of sign-systems inherent in all technologies and used in sign-making activities. Transitions provide insight on how sign-systems move between the multimodal layers in use (Kaptelinin and Nardi, 2009). Technologies and their properties are therefore backgrounded or foregrounded in use (van Leeuwen, 2005), which entails various attentions paid to their affordances and meaning potentials, which in turn affect meaning-making (Ravelli and van Leeuwen, 2018). Transitions are implicitly recognized by previous researchers as a common phenomenon but have not yet been explored in-depth.
This review shows that transitions exist in relation to both modes of representation and properties, and they move between technologies and users. Examples are students' use of visual software, where the tertiary function to manipulate the visual display by semiotic properties such as colours and layout was used by teachers to organize content on a secondary level. In relation to hardware, the affordances of the screen to display content related to the primary function in the use of PCs and the tertiary function in the use of IWBs. More specifically, this means that the offline shape and size of the screen related to the online displaying function in hardware, or the online highlighting functions in software, to convey offline meaning in the use of colours and layout (Wartofsky, 1973) were all foregrounded signsystems and conveyers of meaning in the use of certain technologies, and overlooked in the use of others. These findings contribute to an understanding of the complexity of signmaking in the use of technologies and why technologies can support diverse actions and signmaking activities in some cases and for some users but not others.

Conclusion and implications
Mapping technology use by focussing on the multimodal layers can provide comprehensive insights into traceshow different sign-systems are engraved in combinations of technologies, drawn upon variously by individuals, and therefore lead to variations in Understanding the use of technology actions and sign-making activities. The multimodal layers approach can also contribute to clues regarding what role transitions play in representational signs between technologies, levels of functions and properties, and how such shifts can be understood when traces are linked to individuals' choices of resources to make meaning. The importance of combining technologies' inherent properties to the patterns of human action and thinking will help us to understand how technology use relates to sign-making activities (Jewitt, 2017).
The shifts in functional and semiotic properties between technologies need to be addressed when selecting appropriate technologies (Vigild Poulsen, 2018). The properties are multifaceted, and people make different choices regarding foregrounding or backgrounding information (van Leeuwen, 2005), therefore contributing to variations in meaning-making by the users. This also means that hardware and software technologies are sectioned in relation to the multimodal layers and to the user's need to represent meaning.
Evidence has also been provided that the primary functional level is more commonly used in teaching and learning. Wartofsky (1973) and Adami (2014) pointed out that the solitary use of primary functions could be problematic because it promotes actions rather than meaningmaking. Hence, it is important to consider the role of primary functions in a learning situation, as they have slight relations to the other layers and poor connections to sign-making activities. Thus, teaching and learning undertaken through technology use from mainly a primary function is troublesome, why further explorations are needed on how transitions between technologies' functional properties can give clues to other kinds of multiple use.