MOBILE LEARNING ANALYTICS IN HIGHER EDUCATION : USABILITY TESTING AND EVALUATION OF AN APP PROTOTYPE

This study aims to test the usability of MyLA (My Learning Analytics), an application for students at two German universities: The Cooperative State University Mannheim and University of Mannheim. The participating universities focus on the support of personalized and self-regulated learning. MyLA collects data such as learning behavior and strategies as well as personality traits. This paper presents the findings of a usability test of the web app prototype. A total of 105 students from both universities participated in the study. In addition to a quantitative usability survey, the app navigation and design was evaluated through an eye tracking investigation with seven participants. The findings indicate that the MyLA prototype is easy to use but requires slight modifications concerning the app design.


Introduction
The utilization of digital technologies in everyday life is constantly growing.For students, digital technologies are mostly indispensable.For example, 95 per cent of 14-to 29-year-old Germans used a smartphone in 2016 (Statista/Bitkom, 2017).However, the potential of mobile devices has not been fully developed in universities.There are numerous possibilities how digital technologies can improve learning in higher education institutions.For example, in the NMC Horizon Report 2017, six key trends were identified to adopt technologies in the higher education sector: (1) advancing cultures of learning (higher involvement in innovation development processes); (2) deeper learning approaches (connection of learning with the real world); (3) growing focus on measuring learning (analytics of data out of learning environments); (4) redesigning learning spaces (improvement of the technical infrastructure); (5) blended learning designs (combination of online and face-to-face learning); and (6) collaborative learning (social interaction and intercultural experiences) (Adams Becker et al., 2017).The presented study covers two emerging fields of research in higher education: (1) learning analytics; and (2) mobile learning.
Learning analytics (LA) use static and dynamic information for real-time support of students' learning processes and optimization of learning environments (Ifenthaler, 2015).Besides its flexibility, the main advantages of LA are personalization and the real-time availability of data (Ifenthaler et al., 2014).Lecturers may use rich data for pedagogical decision-making, understand individual performance development of students, identify potential lack of students' capabilities or the need for curricular improvements (Mattingly et al., 2012).With LA, both students and lecturers can reflect on and improve their communication skills.By capturing, analyzing and visualizing the available information about learning and teaching, lecturers are able to make more reliable predictions about their students' academic success (Macfadyen and Dawson, 2012;Mah and Ifenthaler, 2018).Furthermore, students at risk can be identified and given support through personalized pedagogical interventions (Lockyer et al., 2013).Successful applications of LA at universities are, for example, Course Signals at Purdue University, aiming to identify students at risk using an approach similar to a traffic light system (greenno risk, yellowpotential risk, redrisky).Students and lecturers can identify needs for action to improve their learning situation.Furthermore, lecturers are able to intervene and help early (Ifenthaler and Schumacher, 2016).At the University of Wollongong, SNAPP (social networks adapting pedagogical practice) is used (Dawson et al., 2011).The main purpose of this system is to increase collaborative learning.For example, conversations in forums are analyzed for investigating the relationship between students and excluded students can be determined.If students have a high learning orientation, they are discussing more frequently in forums about learning and resource sharing.And, students with a high focus on performance are discussing for example about assessments.The interaction delivers data that can be used to get an insight of student's engagement.At the end of a lecture, the system can also be used for reflection (Sclater et al., 2016).Recent developments include LeAP (LA profiles) which provides students support toward specific learning outcomes in formal learning environments (Schön and Ifenthaler, 2018).

Mobile learning analytics
Mobile learningm-learningenables learning through personal portable electronic devices across multiple contexts (Delcker et al., 2018;Sampson et al., 2013).Through the use of mobile devices, students have access to learning artifacts more easily.The precondition is the availability of Web-connected devices.In addition, students are independent from locations, time, and they can communicate asynchronously (Lin et al., 2016).Mobile learning supports self-regulated learning (Ifenthaler and Lehmann, 2012) on the one hand, and it is an essential element of blended learning environments on the other hand (Al Saleh and Bhat, 2015).An example for a German mobile application is ARSnova of the THM University of Applied Sciences Gießen.ARS stands for an audience response system.Via this app, students and lecturers can communicate interactively.For example, students can ask questions anonymously during a lecture and the lecturer is able to answer the questions in near real time.Moreover, the communication can take place before or after a lecture.Another function is the evaluation of a lecture directly in the app (ARSnova, 2015).
The app MyLA (My Learning Analytics) of the Cooperative State University Mannheim and University of Mannheim targets to improve learning processes in higher education institutions.MyLA provides ubiquitous communication in form of short messages from students to their lecturer and vice versa.This is especially useful for dualsystem courses where students are often away from the campus.Using such data, lecturers can adapt their lectures at university and implement personalized interventions, e.g.adaption of learning materials or a detailed view at topics the students have problems with.Furthermore, the app supports a more personalized and individual way of learning.The students can document their learning motivation through their learning process.The app provides information which enables students to observe their personal progress over time.In summary, MyLA combines LA and m-learning with the extension of personalized learning elements.
The remaining article focuses on the components of the MyLA app prototype and the usability testing of the app.Additionally, the current version of the MyLA app and the MyLA dashboard will be presented.

Literature review
Digital media and new technologies have been deployed frequently in the classroom since the 1980s.With the development of wireless internet access, innovative forms of communication and trans-regional collaboration were born.Digitalization and globalization also led to a change in higher education institutions.Particularly, the shift from traditional teaching (i.e.face-to-face teaching) to adaptive and self-regulated learning (SRL) introduced the need for smart learning classrooms.Hence, the benefits for students are a more personalized and independent learning environment (flexibility of time and location) (Hwang, 2014).According to Koper (2014) improve the teaching quality (Ifenthaler et al., 2018a); support the learning engaging and motivation (Schumacher and Ifenthaler, 2018a); and encourage self-control (Sclater et al., 2016), as well as having potential to support assessment (Ifenthaler et al., 2018b).
The first aspect involves that a lecture can adapt the curriculum toward the students' requirements.As a predictive tool, LA can help to enhance the students' retention.Furthermore, the learners can use the tracked, analyzed and often visualized data as a selfcontrol mechanism.
A current study investigated three relevant features of LA from the students' point of view (Schumacher and Ifenthaler, 2018b): Possibility of self-assessment.Recommendations for learning.Timeline for the evaluation of the students' status quo.
Moreover, privacy is an important aspect when it comes to LA. Students recommend clear transparency and trustfulness of LA applications (Ifenthaler and Tracey, 2016).By providing the users with access to and control over their personal data, confidence can be enhanced (Prinsloo and Slade, 2015).One further aspect is the possibility of making an interpretation of the learners' information very simple.By visualization of the data, learners, lecturers, as well as other stakeholders can easily identify a good or poor performer (Ebner et al., 2015).Furthermore, individual information can be collected from different sources.While login information and the frequency of particular websites are classically representing quantitative data sources, entries in forums or blog, for example, have to be interpreted as qualitative resources (Ifenthaler and Schumacher, 2016).
Dashboards including vast amounts of data, for example, performance or user data, are a technical solution to support the teaching and learning process.The majority of dashboards used in the context of learning and teaching consider four stages.First, teachers are getting aware of several activities (e.g. the use of learning materials) in the course by using a dashboard.Next, they can reflect concerning their actual teaching process under consideration of the collected data.As a third step, teachers can identify at-risk students or isolated ones more easily.Last but not least, dashboards can be used to become aware of the current impact, and thereby how the information can help to support the students, e.g. through proving opportunities for re-socialization or providing feedback to poor performers (Verbert et al., 2013).
Usability testing for mobile applications considers different factors.For example, Nielsen (1993) mentioned five attributes: learnability, efficiency, memorability, errors and satisfaction.Learnability covers an easy-to-learn approach, efficiency leads to high productivity, memorability helps to remember the system more easily, errors should not occur or can be revoked and satisfaction means that users like the system.But there are more attributes that can be valuable for usability testing.As an example, the MAUEM (mobile application usability evaluation metrics) model has nine attributes.The five of Nielsen and four added attributes: effectiveness, cognitive workload, interruptability and simplicity (Saleh et al., 2017).
A usability example will be introduced of ME2.0 (Mobile Electronic Personality version 2).This is an application that helps users to manage their personalization attributes, e.g. against identity thefts.The methodology covered two studies.In the first study, users (N = 9) read different scenario uses of the application before they were interviewed.In a last step, the Mobile learning analytics participants illustrated their own desired UI and navigation.The second study consisted of an online survey (N = 16) and an evaluation of an illustrated prototype (N = 4) (Oyomno et al., 2013).Another example is EASEL (Education through Application-Supported Experiential Learning).This is a mobile platform, which can be used by instructors to send reflection, as well as content prompts.After a pre-conducted needs analysis, a prototype was built.The participants (N = 14) were divided in faculty members (N = 8) and students (N = 6).Each group had to solve similar (but different) tasks toward a hypothetical scenario.The screen was recorded for analyzing how participants navigated within the application, and the facilitator observed them.All participants were asked to "talk-aloud" while completing the tasks.For documenting comments or gestures, cameras were used to record all sessions.After each task, the participants answered short questionnaires about their experiences.Furthermore, the usability questionnaire was conducted with questions concerning their general use of technology and application-related questions.In addition, all participants were asked to discuss their own experiences with the facilitator (Schnepp and Rogers, 2017).

Usability testing
This section focusses on the methodology, as well as on the findings the of MyLA prototype's usability testing.This is the basis for the actual app and the additional MyLA dashboard, which will be introduced in Section 4.

App prototype
The Web app prototype of MyLA consists of three different main units (see Figure 1): (1) My Profile; (2) My Learning; and (3) My Progress.
The general page structure consists of the following: Header: with an icon for the side menu, the MyLA logo and an icon for the home button (except on the index page).
Main: with contents (e.g.select lists in the profile data or pinboard elements like button and post).
Footer: with the linked Cooperative State University Mannheim logo, imprint and privacy, icons linked to: contact, frequently asked questions (FAQ), settings and search function.
My Profile contains two subcategories: (1) profile data with input options like username or university; and (2) the trophy center where app users have access to their rewards, i.e. when entering the profile data or participating on a certain questionnaire.This section includes administrative information of every app user.
My Learning contains two subcategories: (1) the pinboard where students can create posts for their lecturers; and (2) the survey center where students can respond to regular conducted questionnaires.Within this section of the app, the students can communicate with their lecturers and vice versa.Via pinboard, students can post messages by using tags (e.g.ask questions or point to a problem).This will be only visible for the responsible lecturer on their dashboard interface.
My Progress contains two subcategories: (1) MyLA data where students can enter personal data like learning motivation or learning effort; and (2) MyLA stats where the MyLA data will be visualized in charts.This part displays the individual progress in additional (learning) factors.
The design of the app was realized to accommodate the corporate design of the Cooperative State University Mannheim and University of Mannheim.Therefore, one significant color of each education institute had been extracted.As a next step, the colors were combined and supplemented by neutral colors.
According to the approach of LA, the first prototype of the MyLA app was designed to capture user data.Further steps will be collecting the data reports and deriving individual actions for students.Thereby, the main objective of developing personalized and adapted learning environments will be striven.The current app version has been modified and adapted.Additionally, a dashboard for lecturers has been developed.

Research questions
The usability testing focused on three major research questions: RQ1.How intuitive is the MyLA prototype (design, navigation) for students?
RQ2.Is there room for improvement for the development of the MyLA prototype?
RQ3.How can the empirical results (quantitative and qualitative) help to optimize the MyLA prototype for its initial implementation?

Mobile learning analytics
The questionnaire was conducted to evaluate design, navigation, text elements and used icons of MyLA.Beforehand, the students were shortly introduced to the topic of LA.Afterward, they were able to view the MyLA app either on web browser (Cooperative State University Mannheim) or on mobile browser (University of Mannheim), and afterward responded to an online questionnaire.For the purpose of comparability, none of the participants has used the MyLA Web app before.).The majority of the respondents (N = 99) were studying in the field of business administration.More than half of the students were enrolled in a bachelors program (53 per cent) and 47 per cent were studying in a master's program.In addition, seven students (four female, three male) took part in an additional eye tracking study.On an average, participants reported that they are spending 29 days (M = 28.90,SD = 4.56) per month with apps generally, but only four days per month (M = 3.65, SD = 6.06) with apps for learning.
3.3.2Design and procedure.The usability test was divided into two parts using a standardized instrument (see Section 3.3.3):First, the participants made themselves familiar with the MyLA app prototype via web browser or mobile browser.This included the navigation through the app and reviewing the design and the app's structure.Second, they responded to an online questionnaire which was structured as follows: Socio-demographic information, general usage of mobile devices and technologies, and MyLA-specific questions (open and closed questions).MyLA-specific questions focused on navigation and navigation elements, design and app structure.The main group (N = 98) followed this procedure.A smaller group (N = 7) participated in an eye tracking study (see Section 3.3.4).A significant difference between the participants was the device on which MyLA was tested: One group (Cooperative State University Mannheim) tested on a web browser via a personal computer and another group (University of Mannheim) on a mobile browser via a tablet.To ensure anonymity, various identification numbers had been given to the students.
3.3.3Usability instrument.The feedback of the students was committed via an online questionnaire.The question pool within the MyLA-specific part was chosen following the usability testing instrument developed for HIMATT (highly integrated model assessment technology and tools).The instrument has been successfully tested for reliability and validity (Pirnay-Dummer et al., 2010).For the MyLA usability testing, 13 items had been chosen.To give an example, one item was: "I found it easy to navigate through the software".All questions were answered on a five-point Likert scale ranging from highly agree (5) to highly disagree (1). Figure 2 shows the 13 items used in the usability testing.
3.3.4Eye tracking.According to Rayner (2009), eye movements are connected to a participant's attention, and can therefore contribute to the usability testing of screen-based applications.Eye tracking is a standard methodology to record the eye movement of participants.The data evaluation is conducted with special eye tracking software.A very useful feature is the report function that visualizes the eye movements through heat maps or gaze plots (Kurzhals et al., 2017).Common observations in eye tracking studies include (Ehmke and Wilson, 2007): fixation points: where participants have a long look; first look: where participants look first; and non-looking: elements participants do not pay attention to.

ITSE 15,4
After a short introduction, the students were instructed to solve three tasks by using the app prototype.The difficulty of each task was ascending.Therefore, seven AOI (areas of interests) were defined for the analysis with Tobii Pro Studio software.All participants were recorded with regard to their "first look" and on which MyLA contents they looked more often ("fixation points").Furthermore, the student's solution approach was analyzed.The topics of the three tasks can be summarized as follows: (1) Calling the Cooperative State University Mannheim website as soon as possible.
(2) Selecting and saving their respective university in the subcategory profile data.
(3) Creating a post for a lecturer.
Students from the eye tracking study also participated in the survey-based usability test.

Results
The presentation of findings is divided into the survey-based and eye tracking usability test.Data analysis was conducted using IBM SPSS 23.0 and Tobii Pro Studio.

Survey-based usability test.
The main part of the questionnaire was to investigate the app's navigation and design.Figure 2 shows the findings of the 13 items from the survey-based usability test.
The bar charts in Figure 2 are divided into two sections.The chart on left side shows the results concerning the navigation and structure of the MyLA prototype.The second chart on right side highlights the outcomes regarding the design and colors of MyLA.According to the navigation and structure, it is conspicuous that all six average values are constantly high.The highest value was reported for the simplicity of MyLA app prototype with an average of 4.45 (SD = 0.83).Followed by "I found it easy to navigate through the app" (M = 4.36, SD = 0.82) and "The navigation of the app is user-friendly" (M = 4.28, SD = 0.85).Based on the second chart, it is obvious that there were some divergent opinions concerning the design and colors of MyLA.The lowest rated value was the use of color with an average of 3.28 (SD = 1.15).In addition, the participants ranked the design of MyLA as "optically appealing" with 3.36 (SD = 1.19).Mobile learning analytics

MyLA app and MyLA dashboard
In this section, the current applications (MyLA app and MyLA dashboard) will be introduced.The navigation and design were adjusted on basis of the conducted usability testing in spring 2017.

MyLA app
The app prototype of MyLA has been extended and modified in functions and design.While the app prototype was already assessed as "intuitive" and clear in its navigation, only the color scheme was modified.By realizing app version 1.0, this issue was fixed and a more reduced, neutral and clear color scheme was implemented.The three main categories My Profile, My Learning and My Progress are still the same as in MyLA's prototype.Besides the style adjustments of MyLA, the backend infrastructure has been developed.And because MyLA app is primary for students, there was an additional need to create an additional surface for lecturers.Therefore, the MyLA dashboard has been developed in line to the MyLA app.The current app version 1.1 (February 2018) has come up with some slide modifications.Students can receive lecture messages, published only to this lecture (compare to Pinboard Entries in the next section).Moreover, the app user can see what is new in the app via a notification bar, added on top of the footer.The number of notifications is also shown near the home button.Furthermore, it is visible in the side menu and on the app's homepage partitioned by category and subcategory.Additionally, general statistics of the app user can be accessed via the site app statistics.For example, users can see their registration date or the total number of submitted pinboard entries.
Figure 4 shows a screenshot of the MyLA app's homepage consisting of three links (topdown: My Profile, My Learning, My Learning Progress) to all three categories, mentioned Mobile learning analytics above.On the left side, the side menu is visible and displays all categories and subcategories (top-down: My Profile -Profile data and Trophies, My Learning -Pinboard and Survey, My Learning Progress -MyTracker and MyTracker-Stats, My App Statistics).Furthermore, above the application's footer, the notification bar is visible ("1 neue Nachricht" means 1 new notification).

MyLA dashboard
The dashboard version 1.1 has identical functionalities and navigation parts as the application.The dashboard's homepage contains: Pinboard Entries, Survey Center and LectureTracker.The connection between both applications is managed via profile settings.The instructor has to register a certain lecture and define an access code.This code and the automatically given lecture number are perquisites for joining one particular virtual lecture.Via Pinboard Entries lecturers can receive messages, answer them or mark them as read.Additionally, he or she can filter the entries regarding several tags (e.g.question or problem).Furthermore, instructors are capable to send lecture messages consisting texts and optional links and files.Via the feature Survey Center, lecturers have the possibility to create surveys using existent questions or adding own questions.Because of the LectureTracker, instructors can observe the aggregated MyTracker values of the students.There are three options to filter the data: on a daily, weekly or monthly level.The dashboard is only managed by the lecturer to start an action (e.g.new survey), react on students' action (e.g.comment pinboard entry) or observe students' progress (e.g.LectureTracker).
Figure 5 shows a screenshot of the MyLA dashboard's homepage consisting of three grids with all parts and functionalities, mentioned above.On the left side are the Pinboard Entries ("Pinnwand-Einträge") and also a collapsible unit to submit a message to the respective lecture ("Kursnachricht erstellen").In the middle grid, the Survey Center ("Umfrage-Center") is visible.The two collapsible units display a series of default questions ("Übersicht Fragen-Pool") and a form to submit own questions with own options ("Eigene Frage zum Pool hinzufügen").On the right side, instructors can observe the aggregated values of the whole lecture using the LectureTracker ("KursTracker").All MyTracker data, which will be reported by students, result in the average values.Furthermore, above the dashboard's footer, the notification bar is visible identically to the MyLA app.The header contains several buttons like: profile, reload and options button (with e.g.imprint and privacy).

Discussion and conclusion
Usability testing is a very useful and advantageous method for formative evaluation of the development process (Pirnay-Dummer et al., 2010).The project MyLA can benefit from the valuable input of the potential app user.
With regard to RQ1 ("How intuitive is the MyLA prototype (design, navigation) for students?"), it can be suggested that the prototype is intuitive for the target group.Overall, the results were predominantly positive (all average values were higher than 3); however, some issues were identified for improvement, especially with regard to colors.
In addition, students provided feedback within the scope of open questions.The students evaluated the app's clarity, simplicity, navigation/structure and features as very positive.Regarding the assessment of the color scheme, the responses were heterogeneous.Critical issues were partly used dark colors, as well as color combinations.Moreover, some students mentioned that the app contained a broad color spectrum.Furthermore, they suggested additional features for future app versions, e.g. a calendar function.With regard to RQ2 ("Is there room for improvement for the development of MyLA prototype?"), it can be summarized that the colors need to be adjusted.
To answer the RQ3 ("How can the empirical results (quantitative and qualitative) help to optimize the MyLA prototype for the main survey?"), the following aspects can be summarized.The involvement of the target group (students) was very important at this early project stage.The reason for that is very simple, because the students are prospective users of MyLA.Therefore, it is inevitable to get them highly involved.The success of a project depends on its acceptance.If the acceptance is high, the potential usage can be high, too.For reaching a large consumption of MyLA, it is necessary to implement the students' feedback and recommendations.With help of these new insights, future adaptions can be managed.Hence, the findings of the MyLA usability testing provided detailed insights to optimize the app prototype.Some lessons learned of the MyLA usability testing were the following proven statements: The handling of the MyLA Web app prototype is intuitive.The app's structure is easy to learn.The navigation within the app is clear and user-friendly.The students mostly like the idea of MyLA.
The design and colors can be improved, because the opinions deviate fairly high.
With respect to the usability testing and the students' recommendations, the app was adjusted.Furthermore, an additional application for lecturers was developed (MyLA dashboard).For guaranteeing the functionality of both tools, they were tested with a small sample.One aim of the research project is to focus on the needs of the major target groups (students and lecturers); thus, several workshops were conducted in 2017 and further events are planned for 2018.With respect to the current status of the project, it has to be mentioned that the panel study started successfully at the beginning of 2018.First prefunded results of the study will be available in summer 2018.
MyLA can be used in different learning settings like in lectures, workshops or practical phases.The lecturer can integrate both applications as they fit hihe/sher needs the best.There are no universal approaches, but it is highly recommended to clarify the general setup Mobile learning analytics of using MyLA to the students in an early process stage.The applications can be used for student feedback, communication with the lecturer, (flexible) time management and for performance or lecture evaluations.Via the MyLA dashboard, the lecturer can manage these functionalities and observe the incoming data.Additionally, the identification of further functionalities needed by the students is an evident part of the project.
According to the literature, the research area of LA is still in its infancy in Germanspeaking countries (Ifenthaler and Drachsler, 2018;Ifenthaler and Schumacher, 2016).Moreover, with the advance of digital technologies, new skills and competencies are required from students and teachers in higher education institutions, as well as their future workplace (Gibson and Ifenthaler, 2017).In addition, the situation of non-traditional academics will be considered within the project.The combination of LA and m-learning seems to be beneficial to contribute to the individual demands of a diverse group of learners.For example, the project team aims to identify patterns of learning behavior and the management of learning tasks, given a high demand of workload at the university and the workplace (Ifenthaler, 2018).In summary, the project can be seen as an important contribution to the topic of LA for higher education in Germany.
Some limitations and challenges of MyLA (and probably other applications) are as follows: Re-designing and implementation of digital tools in curricula.
Conviction of lecturers to sustainably use digital offers.Boost of m-learning concepts in higher education.
Some chances of MyLA are as follows: Data collection concerning application usage for research improvement in Germany.Privacy-based approach.Data collection for lecturers to identify certain problems within a lecture.
Transfer the Web app to a hybrid app for using device capabilities in future.
As a conclusion of all mentioned points above, there is more research necessary.Some useful research approaches could be: how should a privacy-based application look like in the opinion of students and lecturers, what are overlapping interests of students and lecturers to strengthen and adapt app functions or digital innovations, how can different digital approaches (in one university or cross-university) work together to not "reinvent the wheel".Furthermore, as already discussed, the additional app features have to be implemented in case of further development.All in all, the project delivers a profound base with many important research questions and approaches for future work.
Mobile learning analytics Figure 1.Web app prototype MyLA (translated from German; state April, 2017; own figure)

Figure 2 .
Figure 2. Bar charts with the results of the MyLAspecific questions (own figure)

Figure 4 .
Figure 4. Screenshot of the MyLA app in German (state: February 2018; own figure) Figure 5. Screenshot of the MyLA dashboard in German (state: February 2018; own figure)