An augmented reality-based system for improving quality of services operations: a study of educational institutes

Abstract

Purpose

This study aims to evaluate the usability of the augmented reality-based Evoke Education System (EES) to improve service operations in educational settings. The EES uses an animated character (Moe) to interact with children in a classroom by reproducing their teacher's movements and speech.

Design/methodology/approach

This study uses a quantitative approach for the system usability evaluation. The ESS was evaluated by 71 children aged 6–8 years old, from two primary schools. After interacting with the EES, they completed a system usability questionnaire and participated in a knowledge acquisition test.

Findings

The knowledge acquisition test undertaken on the initial day showed statistically significant improvements for children taught with the EES, compared to children taught through traditional teaching approaches. However, the retest nine days later was not statistically significant (as only one school participated) due to low power. This study used confirmatory factor analysis (CFA), resulting in the identification of five essential factors (likeability, interactiveness, retention, effectiveness/attractiveness and satisfaction) that contribute to the EES's usability. The comparison with existing literature shows that these factors are consistent with the definition of system usability provided by the International Organization for Standardization and current academic literature in this field.

Research limitations/implications

The findings presented in this study are based on the data from only two schools. The research can be extended by involving children from a greater number of schools. Mixed methods and qualitative research approaches can be used for future research in this area to generalise the results.

Originality/value

This study proposes an innovative augmented reality-based education system to help teachers deliver their key messages to the children in a fun way that can potentially increase their knowledge retention.

Citation

Egaji, O.A., Asghar, I., Griffiths, M.G. and Hinton, D. (2022), "An augmented reality-based system for improving quality of services operations: a study of educational institutes", The TQM Journal, Vol. 34 No. 2, pp. 330-354. https://doi.org/10.1108/TQM-07-2021-0218

1. Introduction

Recent research has described that there is a lack of interest and motivation among students towards traditional teaching approaches due to the increasing gap between teaching practice and the 21st-century technological environments (Perez-Lopez and Contero, 2013). This gap has been a significant concern among educational institutions (Di̇slen et al., 2013). Several experts in the field of pedagogy agree that the integration of technology into the learning process is helpful, meaningful and necessary for schools. However, there is a reluctance among teachers to adopt this change (Francis, 2017). According to Ibrahim and Al-Sahara (2007), educators' core focus is to increase students' retention and achievement. Hence, teachers must be willing to adapt their current teaching approaches by incorporating continually changing technology to achieve these goals. This will increase their students' interest and motivation and encourage active learning, thus enhancing their learning outcomes (Gibson, 2001). Research has shown that engaged students tend to be attentive and show positive emotion and demonstrate more effort towards their study (Ibrahim and Al-Shara, 2007). Additionally, student engagement has been associated with the positive student experience, higher grades and fewer dropouts (Connell et al., 1995).

The application of augmented reality (AR)-based classroom learning has the potential to assist students in their learning activities (Klopfer et al., 2009). AR is a potentially game-changing technology; its ability to enhance reality with computer-generated sights, sounds and data transforms the way we view and interact with the world. Literature shows that AR can strengthen students' motivation for learning new things and enhance their educational realism-based practices. AR is “a technology that superimposes a computer-generated image on a user's view of the real world, thus providing a composite view” (Walsh, 2011). It bridges the gap between virtual reality and the real world, and it is popularly used in the military, visual arts, commerce, archaeology, navigation, architecture and in medical and flight training (Chang et al., 2010). There has been an increasing body of research over the last few years on the applicability of AR to education, yet the challenges associated with AR's integration with traditional learning methods, its development and maintenance costs and resistance to new technology still exist (Lee, 2012). There are emerging interactive AR classroom applications for homework, mini-lessons, book reviews, yearbooks, lab safety awareness, deaf and hard of hearing sign language flashcards and so forth (Team, 2017). However, most of the previously developed interactive AR applications are not real-time; they simply utilise pre-stored contents from databases. In addition, little of the work is targeted at children under 12 years old.

Therefore, this study proposes a real-time, cost-effective, easily maintained application called the Evoke Education System (EES) for delivering high-impact lessons to primary school children. The EES utilised an avatar, which gives the children the illusion of speaking to an animated character, Moe (a monkey). The character uses motion capture technology and lip-sync and voice-altering software to mimic their real teacher, who controls it in real-time. The authors chose an animated character that the children are more likely to relate to regardless of gender and ethnicity. According to Theng and Aung (2012), a gender-biased animated character can influence children's interaction with an avatar (this implies the children interact more with the avatar of the same gender). Using the EES, the adult taking on the Moe character can interact with the children as they watch, listen and ask questions. The EES aims to make learning fun, allowing children to feel more comfortable communicating with the avatar rather than interacting with an adult. The authors anticipate that adopting the EES in primary schools will improve the students' interest and motivation. This study evaluates the impact of using the EES as a teaching and learning tool over a traditional teaching approach.

The empirical analysis performed in this study investigates the EES’ potential in educational settings. A confirmatory factor analysis (CFA) was performed to test whether the measure of construct used in this study is consistent with the system usability definition by following the process from Brown (2015). The five factors originated from this study are found to be consistent with the definition of system usability.

This paper contains five sections. Section 2 includes a review of the literature relevant to this study. Section 3 describes the material and methodology. Section 4 presents the results and analysis. Finally, the conclusion and future work are given in Section 5.

2. Related work

2.1 Use of technology in education

In the context of technological change, British education policies placed schools at the top for being the places for innovation and transformation (Williamson, 2012). According to Williamson, information and communication technologies (ICT) can be used in formal education (Williamson and FACER, 2004). The use of digital technologies for information exchange and knowledge generation is also supported by Kitchin and Dodge (2011). The Department of Education established an “EdTech Strategy” worth £10 million in 2019 to develop innovative and technological solutions to overcome teaching and interaction challenges in school education (Education, 2019). Consequently, education technologies “edtech” has developed into a progressive research field (Williamson, 2021b). Recently, the Covid-19 pandemic shifted entire classrooms and campuses to focus on online education through digital media. Millions of school children are being taught through digital technologies (Williamson, 2021a). However, two points of view still exist: pro-digital education and anti-digital education transformation. As it is complex to shift all education to digital mediums, there is a need to find a balance between school practices and digital technologies used for education (Castañeda and Williamson, 2021).

Williamson and Facer argued that computer games and communication technologies could effectively improve children's learning through expert talks, text and digital artefacts in school settings (Williamson and FACER, 2004). Hence, recent studies have focused on exploring the use of digital technologies to promote new ways of exchanging information between teachers and students and to generate knowledge (Kitchin and Dodge, 2011).

2.2 Use of AR and VR to improve quality in education

Among many digital technologies available, AR, VR and human-controlled animation have the potential to provide compelling contextual, on-site learning experiences for children. There has been increasing research interest in the application of avatars for training children in schools. However, scarce literature exists on the application of real-time human-controlled avatars for educating children. An interactive training architecture that utilises remote control avatars and virtual characters was developed by Nagendran et al. (2013); however, no user testing on the platform's efficacy was reported.

Most of the related work in this area has shown a positive effect in adopting an avatar to teach children. A study exploring the evolution of animation for teaching and learning in classroom settings showed a better understanding of the students' subject area when animation was used (Falvo, 2008). Fortier et al. developed a tablet-based animated avatar for pain assessment and intervention in a home setting for children aged 8–18 years. The tablet-based application's key components involve daily pain and symptom diaries to be completed by the children and uploaded via a cloud for remote monitoring. The pilot study, which consists of 12 children, showed increased engagement, and the children were satisfied with the application (Fortier et al., 2016). Other researchers proposed courseware, which allows teachers to script animated pedagogical agents for teaching the English language. Their experimental results showed that the group taught with the courseware outperformed the group taught with the traditional approach (Hong et al., 2014). A similar 3D virtual interactive tool, called Alice, was developed to engage middle school children (10–14 years old) in writing interactive stories for maths, language, history and art courses using templates available in Alice World. The researchers claimed that the teachers and students engaged actively with the Alice virtual tool (Rodger et al., 2009).

Other authors have studied the efficacy of the avatar's expression or design on learning. An example is research carried out by Theng et al. (2012) in which the authors investigated the effect of an avatar's expression on 24 (6–8 years old) children's emotional response and motivation towards learning using Ortony, Clore and Collins (OCC) cognitive theory. The authors claimed the children were affected by the expression of the avatar. In addition, they found a gender bias in the children's interactions with the avatar as they enjoyed interacting more with an avatar of their own gender (Theng and Aung, 2012). The customisation of a game avatar can affect both the subjective feeling of presence and the psychophysiological indicators of emotion during gameplay, making the game experience more enjoyable (Bailey et al., 2009). The use of avatars from children's favourite TV characters can be used as a substitute for CD-ROM or online learning because they are more likely to relate to this character. An interactive avatar has potential application in children's safety education, for example, road safety or other safety lessons (Sheth, 2003).

Avatars have also been employed in the education of children who are deaf or who have speech impairment. The impact of a 3D signing avatar in boosting the learning experience of deaf gamers was investigated. The study consisted of 6 deaf participants (5 boys and 1 girl), aged between 10 and 14 years old. The authors claimed that the interactive avatar could help deaf gamers (Bouzid and Jemni, 2016). Similar works have reported positive feedback, and children's increased interaction with their gaming platform, as a result of interactive avatars (Adamo-Villani et al., 2005; Nasiri et al., 2017).

The use of avatars in education has also found application in training children with autism to develop their social and emotional responses. Previous research has investigated the effect of using a collaborative virtual learning environment (CVLE) 3D animated scenario (empathy system) to enhance the empathy of people with autism spectrum conditions (ASCs) (Cheng et al., 2010). The study was conducted over 5 months, consisting of 3 participants with ASCs. The authors claimed an effect on the participants' understanding and use of empathy within the CVLE 3D empathy system. Konstantinidis et al. evaluated a computer-aided learning platform for children with autism. The platform was evaluated by 13 educators of autistic persons over a period of one month. Their study demonstrated an interactive learning environment's efficacy to develop and facilitate people's learning needs with autism (Konstantinidis et al., 2009).

Teachers' perception of the use of animation in their professional career development and of animation as a medium for student engagement has also been studied (Chan, 2015). Despite the teachers' positive attitudes towards using animation for teaching and learning, they express practical and technical concerns. Their concerns include access to resources and technical know-how in the development of this animation. The use of animation in education positively affects students' attitudes and achievement (Unal-Colak, 2012). However, questions relating to their usability in education, efficiency and compatibility with traditional teaching methods still need answering (Shelton and Hedley, 2004).

Previously, we have conducted a number of studies using VR to teach university and school students different tasks related to health and safety. We also developed a VR-based gas assessments application to teach young gas operatives about the basic education process related to gas assessment procedures. The application was evaluated by 32 gas operatives, and they appreciated the support provided by the VR environment to learn about gas assessment training tasks in a risk-free environment (Asghar et al., 2019). In addition, our research team developed another VR-based application called Motion Rail to teach schoolchildren about the use of railway crossings using scenarios in a VR environment. The application was tested with schoolchildren, and they successfully learnt the process of using level crossings and foot crossings safely (Dando et al., 2018).

2.3 Study hypothesis

The related work shows the emerging research interest in using AR and 3D animated technology to develop innovative course content for students. However, most of the applications presented are not real-time; they utilise pre-stored contents from databases. In addition, most of these studies have not carried out a knowledge acquisition test to evaluate how they increase motivation, develop interest in learning and translate to knowledge retention. An interactive AR animation would contribute to a more learner-centred teaching method (Lauer et al., 2001). Although some authors evaluated AR and 3D animated technologies performance compared to the traditional approach, very few reported their studies' effect size and a system usability evaluation. This study aims to address some of these issues by proposing a novel real-time human-controlled AR application for educating children, called the Evoke Education System (EES). The efficacy of the system, as compared to the traditional approach and system usability, is evaluated. The authors anticipate that adopting the EES to assist primary school teachers will increase students' engagement, motivation and knowledge acquisition compared to the traditional approach. In summary, the study aims to investigate the following hypothesis:

H0.

The differences between the knowledge acquisition test for participants that receive their lessons via the EES and those that receive their lesson via the traditional face-to-face approach is random with no statistical significance.

H1.

The differences between the knowledge acquisition for participants who receive their lesson via the EES and those receiving that lesson via the traditional face-to-face approach are statistically significant.

3. Materials and methods

3.1 Research design

This paper builds on an initial pilot study carried out by Asghar et al. (2018b). During this pilot study, the children and teachers were asked to answer some questions at the end of the testing. The children liked the avatar, Moe, as a teacher and were interested in knowing more about the character and listening to what Moe had to say to them. They were very excited to answer Moe's questions and liked Moe's shape, facial features and body movements. They were also eager to carry out the tasks set by Moe. These tasks increased the children's enthusiasm and they enjoyed the interactivity with Moe. In addition, the teachers were happy with the EES prototype. They found it fun and easy to interact with the children using this animated character. The teachers essentially agreed that the children were able to obtain useful information from Moe during their lessons. As most modern-day teachers are IT literate, they agreed that integrating this current prototype into their teaching would not pose many challenges. However, they recommended that more teacher training would be required (Asghar et al., 2018b). The pilot study results were promising, highlighting the need for this technology in education.

The current research focuses on verifiable observations, and usually results in this type of research are presented numerically (Guba and Lincoln, 1994). A comprehensive literature survey by Hunter and Leahey (2008) has indicated that almost 66% of top research conducted in the last 80 years has used quantitative approaches for such research in all fields. The quantitative research asks participants for their opinion in a structured manner and generates statistics and facts. The use of questionnaires is the most popular medium of data collection for such research studies. Therefore, we adopted a questionnaire-based research methodology for the current study.

3.2 Participants

The experiment's participants consisted of year 1, 2 and 3 pupils from two separate primary schools. The two primary schools were recruited based on their willingness to participate in the research and being within a reasonable driving distance from the research team. The participating class size ranged from 15 to 20 per class. The two selected primary schools were named School A (37 participants) and School B (34 participants). Overall, 71 participants (48 boys and 23 girls) with age groups of six years (23), seven years (47) and eight years (1) were involved in the testing, along with four teachers and four teaching assistants. The children's mean age was 6.69, with a standard deviation (SD) of 0.50. The participants completed the questionnaire and participated in the knowledge acquisition test.

3.3 Description of the EES

The EES consists of a wide range of hardware components across two locations (audience and operator rooms). The system gives the illusion that an animated character is interacting with the audience in real-time. The audience room has the Imagination Station, which enables the viewers to interact with the EES. The teacher will be located in the operator room with the Toybox, which allows the teacher to communicate with the audience via an avatar called Moe. There is also a video and audio feedback link from the audience room to the operator room.

3.3.1 The audience room

The main display cabinet (Imagination Station) is located in the audience room, as shown in Plate 1. Based on this study's scope, the audience will be young children, who it is hoped will engage with the animated character in a more enthusiastic manner than just being talked to by an adult. The cabinet is a custom-built wooden enclosure with hardware consisting of a transparent television display, video camera, HDMI streamer (transmitter), sound bar and artificial decorations comprising flowers, grass and trees. The avatar appears on a transparent white screen, revealing the artificial decoration (flowers, grass and trees). The display cabinet's size can be smaller or bigger, depending on the end-user requirements.

3.3.2 The operator room

The operating room can be smaller than the audience room, requiring only enough space to contain all the operator equipment (Toybox), as shown in Plate 2, and big enough to allow safe movement. The hardware composition of the Toybox includes the main PC, Kinect camera, HDMI streamer (receiver), monitors, wired headphones with microphone and game controller. The hardware display for the operator room is shown in Plate 2. The Kinect 2 camera maps the operator's movements to the on-screen avatar during the performance, and it is connected to the PC via the Kinect for Windows Adapter.

3.3.3 The Moe character design (the programme)

The programme was developed using Unity (a multi-platform 3D game engine). Once launched, it shows a standard Unity control panel, which allows the user to change the display resolution, as well as remap the game controller buttons to the various functions permitted in the programme (Smile, Frown, Reset Face, Raise Curtain, Lower Curtain and Start Performance). The leading interactive part of the application is the setup screen. Once the operator has made the appropriate selections, the start button is enabled. The operator can get into a position visible to the Kinect camera. Once the operator is ready, they can use the game controller to raise the curtain and reveal the avatar on a white background. The operator can view and interact with the audience by speaking into the microphone. Due to the nature of the “transparent” television display, white colours are rendered transparent on the screen; the result is an illusion of the avatar standing at the front of a “box” mimicking the person controlling it with lip-sync and voice-altering software.

3.4 Questionnaire design and validation

A questionnaire-based survey was used to measure the system usability. As the targeted population consisted of young primary school children, appropriate data collection tools were investigated. The relevant literature points out that using 5- or 7-point Likert scales can be difficult for participants within this age group to follow (Mellor and Moore, 2013). Our research team have previously conducted technology usability evaluation studies with children. Through the experiences gained in those studies, we also concluded that using questionnaires with large Likert scales could be challenging for the children (Dando et al., 2018; Asghar et al., 2018b). Therefore, our research team conducted a pilot study with schoolchildren using a 3-point Likert scale, and it showed that it is better to use a 3-point Likert scale data collection tool with primary school children. Therefore, a 3-point Likert scale was developed for this experiment, as shown in Table 1.

Additionally, we improvised with the use of smiley/emoji faces with the Likert scale for the children, and they were very happy while filling in these questionnaires.

The questionnaire was validated through a three-step validation process and reviewed by research colleagues, primary school teachers and an academic psychologist for scope and structure. This process helped to redevelop the questionnaire. The finalised usability questionnaire for the experiment consists of 23 questions, as shown in Table 10.

3.5 Experimental design and data collection

A pre-test/post-test quasi-experimental design (Harris et al., 2006) was selected to demonstrate the authors' hypothesis. The same group served as the control and intervention for ethical reasons. The authors followed the same experimental procedures for the two different schools that participated in this study. Firstly, the system was introduced to the teachers, and they were instructed on how to operate the EES. The participants were gathered in a single class for an introductory session to the EES on the test day. The exchange of pleasantries between the animated character “Moe” and the participants lasted for 10 min before the session ended.

To avoid a potential order effect, the authors counterbalanced the experimental design by dividing participants into two groups (Groups 1 and 2) and presenting the tests in a different order for each group, as suggested by Shaughnessy et al. (2000). The experiment consisted of two sessions – Session 1 followed by Session 2. Two different scripted lessons, Lesson 1 and Lesson 2, of equal academic complexity, were delivered in Sessions 1 and 2, respectively. The teachers were allowed to deliver the lessons in their normal style, which was in an interactive and pupil-focused manner. The theme of the week at School A was “Responsibility”; therefore, both stories taught were on the same theme. Lesson 1 was a story about a character “Koki the frog”, teaching the participants how to develop those fundamental values that constitute a person's character. Lesson 2 was a story about a character “Rosa, the rabbit”, teaching the participants about the importance of being responsible for their behaviour and their belongings.

During Session 1, Group 1 was taught Lesson 1 using the EES, while Group 2 pupils received Lesson 1 in a more traditional style, with a teacher presenting at the front of the classroom. The participants were allowed time to ask questions at the end of the lessons. Afterwards, Group 1 completed a usability questionnaire and knowledge acquisition test based on Lesson 1, while Group 2 only completed a knowledge acquisition test based on Lesson 1. The knowledge acquisition test consisted of 10 questions.

After a short break, the groups swapped for Session 2, and Lesson 2 was taught to both groups. At the end of Session 2, Group 1 only completed the knowledge acquisition test based on Lesson 2, while Group 2 completed the usability questionnaires and the knowledge acquisition test based on Lesson 2. The teachers at both sessions remained the same all through the testing process. A summary of the experimental procedure is shown in Table 2. A second assessment for both Lessons 1 and 2 was carried out nine days after the initial test day.

This procedure was repeated in School B, using the same lessons.

3.6 Ethical considerations

The Faculty of Computing, Engineering and Science Ethics Committee provided the ethical approval for this study in 2018. The schools granted permission to the research team to conduct the experiment on their premises. and data were collected in June and July 2018. All adults and parents/guardians of the children who participated in the test were provided with a detailed information sheet about the experiment, and they provided written informed consent. The children were also asked for their consent.

4. Result and discussion

This section consists of the results and analysis of the knowledge acquisition questions and the application of the CFA to the participants' questionnaire data sets. The knowledge acquisition test helped to answer the authors' research hypothesis, and the CFA explored the crucial factors contributing to the EES's usability.

4.1 Knowledge acquisition test overview (Schools A and B)

The participants' performance in the knowledge acquisition test for both sessions was analysed. A two-tailed paired t-test (t) was performed to check for statistical significance between the mean difference of the EES session and traditional teaching approach. The combined knowledge acquisition assessment for School A and School B participants is shown in Table 3. The results show a significant difference between the EES (Mean = 9.02, SD = 0.85) and the traditional teaching approach (Mean = 8.54, SD = 1.35), t(61) = 2.35, p = 0.02. Hence, the null hypothesis is rejected, and the alternative hypothesis is accepted. The strength or magnitude of this effect can be calculated using Cohen's effect size (Cohen, 1988a). The Cohen's effect size, shown in Table 3, implies a small effect size of 0.43 (two groups differ by 0.43 SD).

Several authors have criticised the lack of statistical power analysis in research planning in behavioural and social science (Cohen, 1988b, 1992). As a result, a two-tailed statistical power for a fixed sample size, at 95% confidence, was calculated (Faul et al., 2007, 2009). As shown in Table 3, the statistical power of 0.91 was calculated, which implies that 91% of the time, there will be a statistically significant difference between the EES and the traditional teaching method. It also means that 9% of the time the experiment is run, the outcome will not show a statistically significant effect between the two teaching approaches, even though there could be one in reality. The generally accepted statistical power is 0.8, but researchers can specify a higher value, depending on the study (Dybå et al., 2006).

Another knowledge acquisition test (10 questions) was carried out nine days after the initial test day. School A failed to carry out the retest in the stipulated period. Consequently, this paper only includes the retest results from School B. The authors' presumption, at this stage, was that the EES's impact on the children's knowledge acquisition would be noticeable. A similar analysis was conducted, retaining the same hypothesis as in Table 3. The retest results are shown in Table 4. The mean score of the retest shows the EES with a slightly higher value than the traditional teaching approach. However, there is an insignificant difference between the EES (Mean = 9.38, SD = 0.80) and the traditional teaching approach (Mean = 9.27, SD = 1.00), t(26) = 0.53, p = 0.60. Hence, the null hypothesis is accepted. The statistically insignificant outcome of the p-value could be due to the insufficient statistical power of 0.10.

The results of the retest indicated improved results for both teaching approaches, which suggests delayed learning. This outcome was similar to that of Perez-Lopez and Contero (2013). However, their findings showed a reduction in the participant's knowledge acquisition and retention score for a traditional learning approach after two weeks, while that for AR increased.

Finally, the authors took care to control potential order effects by staggering the order of the material presented, using a mixed-design ANOVA, analysed as a 2 × 2 repeated measures ANOVA. The level of treatment (instruction methodology) is a within-subject factor. All respondents experienced Moe and their regular teacher. The treatment order is between-subjects where some saw Moe first (first-order), while others saw the regular teacher first (second-order); thus, each respondent saw one of these orders, not both. The null hypothesis might be: H0: μFirst order= μSecond order and an “alternative hypothesis” might be: H1: μFirst order≠ μSecond order. According to the descriptive statistics shown in Table 5, the participants in the first order have an average test score of 8.41, while those in the second-order have an average test score of 8.65. The test of between-subject's effects results, as shown in (Table 6), has an F statistics = 0.612 and p = 0.043 for treatment. The p-value is not statistically significant (p < 0.05); thus, the null hypothesis is accepted. Hence, the mean between the two groups has no statistically significant difference.

4.2 System usability through CFA

System usability is one of the core concepts in human-computer interaction research. There are multiple definitions of usability, and a popular one comes from the ISO (The International Organization for Standardization). According to ISO 9241–11, the usability is “the extent to which specified users can use a product to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use” (ISO, 1998). The application of CFA will help to highlight which important factors contribute towards the usability of the EES. CFA checks whether a measure of the construct is consistent with a researcher's understanding of the nature of that construct (Brown, 2015), which, in our case, is system usability. The first step in performing a CFA is checking the respondents' data suitability for the process by completing the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett's test of sphericity.

4.2.1 KMO measure of sampling adequacy

We performed the KMO measure of sampling adequacy (Kaiser and Rice, 1974) and Bartlett's test of sphericity (Bartlett, 1950) to assess the suitability of the data set for factor analysis. As per Table 7, the KMO value of 0.629 and Bartlett's test of sphericity of p < 0.00 for the data set indicate that sufficient correlation was found within the correlation matrix for factor analysis to proceed according to Tabachnick et al. (2007).

4.2.2 Criteria for factor extraction and retention

This paper used the Kaiser's criterion (eigenvalue >1 rule), the cumulative percentage of variance and the parallel analysis extraction approaches. According to Table 8, a total of eight factors have an eigenvalue >1, with a cumulative percentage variance of 78.938%. Three of the eight factors have less than three variables. Literature suggests that each factor should have at least three variables (Maccallum et al., 1999). Hence, we retained five factors meeting this criterion. The total variance explained by the five factors is 63.448%, which is above the acceptable limit of 40% (Dunteman, 1989).

A parallel analysis was performed to validate the five factors retained in Table 8. The literature recommends the suitability of parallel analysis for determining the total number of factors to extract or retain (Courtney and Gordon, 2013). In a parallel analysis, the eigenvalues are compared to a random order of eigenvalues. Factors that have an actual eigenvalue greater than the ordered eigenvalue from a random matrix are retained (Horn, 1965). As shown in Table 9, only the first five (component numbers 1 to 5) of the eight generated factors are retained.

4.2.3 Factor identification using rotated component matrix

After the parallel analysis, the SPSS model was rerun with five fixed factors. The resulting rotated component matrix is shown in Table 10. The factor-loadings for each survey item is above the acceptable value of 0.40 (Hair et al., 2006), ranging from 0.466 to 0.853. The factor mean shows a strong positive response from the children, as this ranged from 2.70 to 2.87. The global Cronbach's alpha value for the survey is 0.875, and for factors 1, 2, 3, 4 and 5 ranges from 0.68 to 0.87, which indicates internal consistency (Lewis, 2018).

4.2.4 Factor naming and discussion

This section describes the factor names and their composition, and outlines their importance via relevant literature.

F1: System likeability – This factor consists of survey items, which constitute the system likeability measures. The system likeability aspect of any system enhances the usability experience of users using that system (Isip and Caparas, 2018). For the current study, the factor played a vital role in the participants' choice to prefer teaching via the EES for their lessons. Therefore, we can conclude that the system likeability factor contributes positively towards the usability of the EES, with a mean value of 2.79.

F2: Interactiveness – This factor contributes positively to the EES's usability, with a mean value of 2.70. It contains items that mainly constitute interactive measures. However, the variable “I remembered the lessons taught by Moe” also loaded high on this factor. This suggests that the EES' level of interactiveness plays an essential role in the participants' knowledge acquisition. The literature showed that increasing interactions during teaching sessions between students and their teachers improves their achievement and knowledge acquisition (Huang et al., 2018; Fredricks et al., 2004). We validated this by the knowledge acquisition test discussed previously in this paper.

F3: System retention – This factor constitutes retention measures, and it refers to the users' willingness to use the system again in the future (Huang et al., 2016). However, the variables “Moe has a positive and friendly facial expression” and “Moe's teaching method is easy to understand” also loaded highly on this factor. This could mean that Moe's cheerful and friendly facial expression and the ease of understanding Moe's teaching methods impact the participant's willingness to retain the technology. This factor has the highest mean value of 2.87, indicating a positive contribution to the EES' usability. This is consistent with the literature, as the system, which fits to the users' strengths, is usually retained for longer periods of time (Seok and Dacosta, 2014).

F4: Effectiveness/Attractiveness – Effectiveness and attractiveness are essential elements of technology adaptation and retention. Technology effectiveness refers to a system's ability to accomplish its stated purpose and is considered as the primary goal and critical aspect of technology acceptance practices (Asghar et al., 2018a). Many researchers believe that system attractiveness influences its usefulness, enjoyment and ease-of-use and can contribute strongly towards system usability (Armeen et al., 2019; Van Der Heijden, 2003). This factor consists mainly of attractiveness and effectiveness measures. However, the variable “I will recommend Moe to my friends” loaded highly on this factor. This could imply that the participants were willing to recommend the technology to their friends because it was attractive/effective. This factor has a positive influence on system usability, with an overall mean value of 2.73.

F5: Satisfaction – This factor refers to the level of achievement the user can feel through their interaction with the system and how much the system meets their expectations during the learning activities (Mtebe and Raphael, 2018). The variables “I could hear Moe clearly”, “Moe is friendly” and “I like the way Moe teaches” all loaded on this factor. This factor has a positive influence on system usability, with an overall mean value of 2.84. Again, this finding is consistent with existing literature, which indicates that if a system meets users' expectations and benefits them it will improve their satisfaction with the system (Seok and Dacosta, 2014).

In summary, the identified factors are consistent with the ISO definition of usability. Additionally, these factors are compatible with several studies in the existing academic literature (Teran, 2018; Lewis and Sauro, 2009; Gaines et al., 1996; Takayama and Kandogan, 2006; Seok and Dacosta, 2014; Van Der Heijden, 2003; Nehari and Bender, 1978). Research has shown that the use of technologies within schools can enhance students' academic motivation and improve their achievements ( Seok and Dacosta, 2014). The consistency of these factors with existing literature shows that the EES can contribute to positive learning for schoolchildren.

4.3 Managerial implications and study applications

The benefits of adopting the EES for the children, teachers, schools and overall educational operations are enormous. Teachers in primary schools can use the EES remotely as a supplementary tool in teaching children to read. The need for primary schools to adapt their teaching on account of the disruption caused by the Covid-19 pandemic has increased educators' interest in finding appropriate tools that are effective for remote/distance learning. Given that the EES has been shown to engage children, this highlights its potential to be adapted for remote/distance learning. The system also has the potential to enrich the education for children with various types of disability, for example, autism. The most appropriate use of this technology was found to be where high impact is needed. Another exciting application of this technology is its use by the police and social services to make interviewing vulnerable children easier. This is because children are likely to be more comfortable with familiar, animated characters than they are with adults.

The questionnaire designed for this study was based on the actual needs of young schoolchildren, and traditional 5- or 7-point Likert scale questionnaires are too complicated for them to understand. Therefore, we also recommend that future researchers design their data collections tools based on the needs of children, if they are their targeted research participants.

4.4 Study limitations

This study's limitation includes the small sample size and the low response rate for the knowledge acquisition retest. The reason for the small population size was that the target population consisted of young schoolchildren and Covid-19 restrictions prevented accessing further schools to take part in the research activities to extend the study. To generalise the results, more samples that include wider demographics are required. Currently, there are limitations to the number of subjects the technology can teach because there is no multimedia presentation functionality. Hence, further development is being carried out to include this functionality so the system can cover a wide range of subject areas in schools.

5. Conclusion and future work

This paper presents an innovative learning approach for children using the EES, which utilises a human-controlled avatar (Moe) to enhance the children's learning experience in a classroom. EES uses motion capture technology and lip-sync and voice-altering software to mimic the person controlling it. The characters appear on-screen visible to the children, managed by a teacher from another room, and it talks and moves in real-time. A pre-test/post-test quasi-experimental design has been used for this study. In addition, a bespoke 3-point Likert-scale questionnaire that went through a three-staged validation process was developed.

The proposed innovative learning approach performed better in the knowledge acquisition test for the initial and retest day. However, only the initial test day results were statistically significant. The retest nine days later was statistically insignificant because of the small power of 0.10 as only School B participated in the retest. CFA was applied to highlight the factors that contribute to the usability of the EES. Five factors were extracted, and their mean scores support the argument that the EES positively affects the children's learning process.

Further research is needed to understand the effect of this technology's repeated exposure to children, which is currently unknown. The children's interest may reduce over time; alternatively, they may become more comfortable with the system, thus enhancing their learning outcome. Hence, more sessions with schoolchildren will help understand the EES' long-term real impact in assisting children in their education. Although the teachers' completed questionnaires were not included in this study, because the sample size is too small, a previous pilot study found that the teachers are equally excited to adopt the EES for their lessons. The teacher's analysis will be presented in future work when more data have been collated.

Figures

Plate 1

Audience room with children looking at EES

Plate 2

Toybox in the operator room controlled by the teacher

Table 1

Questionnaire Likert scale

Table 2

Testing procedure

Sessions	Activity	EES	Traditional Teaching
Session 1	Lesson 1	Group 1	Group 2
Break
Session 2	Lesson 2	Group 2	Group 1

Table 3

Initial test of knowledge acquisition assessment in Schools A and B

	EES Mean score (std. dev.)	Traditional approach Mean score (std. dev.)	t-test (t)	p	Effect size (d)	Power (1-β)
First assessment	9.02 (0.85)	8.54 (1.35)	2.35	0.02	0.43	0.91

Note(s): The test is out of 10, and the number of participants “n” = 61

Table 4

The retest of knowledge acquisition assessment in School B

	EES Mean score (std. dev.)	Traditional method Mean score (std. dev.)	t-test (t)	p	Effect size (d)	Power (1 -β)
Retest – 9 days later	9.38(0.80)	9.27(1.00)	0.53	0.60	0.13	0.10

Note(s): The test is out of 10, and the number of participants “n” = 26

Table 5

Descriptive statistics

Teach Method	Treatment	Mean	Std. deviation	N
With regular teacher	First order	8.41	1.394	27
	Second order	8.65	1.323	34
	Total	8.54	1.349	61
With-EES	First order	8.96	0.980	27
	Second order	9.06	0.736	34
	Total	9.02	0.846	61

Table 6

Test of between-subjects effect

Source	Type III sum of squares	df	Mean square	F	Sig
Intercept	9257.831	1	9257.831	6687.418	0.000
Treatment	0.847	1	0.847	0.612	0.043
Error	81.678	59	1.384

Table 7

KMO and Bartlett's test of sphericity

Kaiser-Meyer-Olkin measure of sampling adequacy		0.629
Bartlett's test of sphericity	Approx. chi-squared	989.574
	Degrees of freedom	253.000
	Significance	0.000

Table 8

Variance and cumulative percentage

Component	Initial eigenvalues			Extraction sums of squared loadings			Rotation sums of squared loadings
	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %	Total	% of Variance	Cumulative %
1	6.835	29.715	29.715	6.835	29.715	29.715	2.843	12.361	12.361
2	2.319	10.081	39.796	2.319	10.081	39.796	2.808	12.208	24.569
3	2.051	8.918	48.715	2.051	8.918	48.715	2.372	10.314	34.883
4	1.846	8.025	56.740	1.846	8.025	56.740	2.349	10.211	45.095
5	1.543	6.708	63.448	1.543	6.708	63.448	2.307	10.033	55.127
6	1.302	5.663	69.111	1.302	5.663	69.111	2.280	9.913	65.040
7	1.177	5.117	74.228	1.177	5.117	74.228	1.897	8.246	73.286
8	1.083	4.710	78.938	1.083	4.710	78.938	1.300	5.652	78.938

Note(s): Extraction method: a principal component analysis. Note: number of participants “n” = 71

Table 9

Parallel analysis

Component number	Actual eigenvalue from PCA	Eigenvalues of random data matrix	Decision
1	6.83	2.22	Accept
2	2.32	1.99	Accept
3	2.05	1.81	Accept
4	1.85	1.67	Accept
5	1.54	1.54	Accept
6	1.30	1.41	Reject
7	1.18	1.31	Reject
8	1.08	1.23	Reject

Note(s): Number of participants “n” = 71

Table 10

Identifying the underline factors using rotated component matrix

Factors	Survey items	Factor loadings	Factor mean	Standard deviation	Eigenvalue	Percentage of variance	Cumulative % variance	Cronbach's alpha
F1: Likeability	It is easy to learn from Moe	0.844	2.79	0.37	6.84	29.72	29.72	0.87
	I had fun learning from Moe	0.754
	I would like my other lessons to be taught by Moe	0.742
	I found it easy talking to Moe	0.715
	I like Moe as a teacher	0.702
	Moe has a fun way of teaching	0.537
F2: Interactiveness	I had the opportunity to ask Moe questions	0.853	2.70	0.48	2.32	10.08	39.80	0.81
	I remembered the lesson taught by Moe	0.727
	I could talk to Moe clearly	0.664
	I am happy with Moe's answers to my questions	0.632
F3: Retention	I would like Moe to teach my friends	0.803	2.87	0.25	2.05	8.92	48.72	0.74
	Moe has a positive and friendly facial expression	0.759
	I would like to hear more from Moe	0.694
	Moe should regularly teach at schools	0.655
	Moe's teaching method is easy to understand	0.466
F4: Effectiveness/Attractiveness	I liked the way Moe smiled	0.797	2.73	0.36	1.85	8.03	56.74	0.68
	I easily understood all the information given by Moe	0.665
	Moe answered most of my questions	0.548
	I enjoyed my experience with Moe	0.528
	I will recommend Moe to my friends	0.496
F5: Satisfaction	I could hear Moe clearly	0.728	2.84	0.34	1.54	6.71	63.45	0.70
	Moe is friendly	0.703
	I like the way Moe teaches	0.667