Mobile learning analytics in higher education: usability testing and evaluation of an app prototype

Matthias Kuhnel (Learning, Design and Technology, University of Mannheim, Mannheim, Germany)
Luisa Seiler (Cooperative State University Mannheim, Mannheim, Germany)
Andrea Honal (Cooperative State University Mannheim, Mannheim, Germany)
Dirk Ifenthaler (Learning, Design and Technology, University of Mannheim, Mannheim, Germany and UNESCO Deputy Chair of Data Science in Higher Education Learning and Teaching, Curtin University, Bentley, WA, Australia)

Interactive Technology and Smart Education

ISSN: 1741-5659

Publication date: 19 November 2018

Abstract

Purpose

The purpose of the study was to test the usability of the MyLA app prototype by its potential users. Furthermore, the Web app will be introduced in the framework of “Mobile Learning Analytics”, a cooperation project between the Cooperative State University Mannheim and University of Mannheim. The participating universities focus on the support of personalized and self-regulated learning. MyLA collects data such as learning behavior, as well as personality traits. Last but not least, the paper will contribute to the topic of learning analytics and mobile learning in higher education.

Design/methodology

For the empirical investigation, a mixed-method design was chosen. While 105 participants took part in the conducted online survey, after testing the app prototype, seven students joined an additional eye tracking study. For the quantitative part, a selected question pool from HIMATT (highly integrated model assessment technology and tools) instrument was chosen. The eye tracking investigation consisted of three tasks the participants had to solve.

Findings

The findings showed that the students assessed the idea of the app, as well as the navigation positively. Only the color scheme of the prototype was not very attractive to a noticeable amount of the participants. So, it requires slight modifications concerning the app design. For the eye tracking study, it can be stated that the students viewed the relevant parts, and they basically had no difficulties to solve the tasks.

Originality/value

Due to the empirical testing of the app prototype, the project team was able to adjust the application and to add further features. Furthermore, the backend was programmed and an additional tool (MyLA dashboard) was developed for lecturers. A mutual understanding of the targets, privacy issue and relevant features are indispensable for further development of the project.

Keywords

Citation

Kuhnel, M., Seiler, L., Honal, A. and Ifenthaler, D. (2018), "Mobile learning analytics in higher education: usability testing and evaluation of an app prototype", Interactive Technology and Smart Education, Vol. 15 No. 4, pp. 332-347. https://doi.org/10.1108/ITSE-04-2018-0024

Download as .RIS

Publisher

:

Emerald Publishing Limited

Copyright © 2018, Matthias Kuhnel, Luisa Seiler, Andrea Honal and Dirk Ifenthaler.

License

Published by Emerald Group Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode


1. Introduction

The utilization of digital technologies in everyday life is constantly growing. For students, digital technologies are mostly indispensable. For example, 95 per cent of 14- to 29-year-old Germans used a smartphone in 2016 (Statista/Bitkom, 2017). However, the potential of mobile devices has not been fully developed in universities. There are numerous possibilities how digital technologies can improve learning in higher education institutions. For example, in the NMC Horizon Report 2017, six key trends were identified to adopt technologies in the higher education sector:

  1. advancing cultures of learning (higher involvement in innovation development processes);

  2. deeper learning approaches (connection of learning with the real world);

  3. growing focus on measuring learning (analytics of data out of learning environments);

  4. redesigning learning spaces (improvement of the technical infrastructure);

  5. blended learning designs (combination of online and face-to-face learning); and

  6. collaborative learning (social interaction and intercultural experiences) (Adams Becker et al., 2017).

The presented study covers two emerging fields of research in higher education:

  1. learning analytics; and

  2. mobile learning.

Learning analytics (LA) use static and dynamic information for real-time support of students’ learning processes and optimization of learning environments (Ifenthaler, 2015). Besides its flexibility, the main advantages of LA are personalization and the real-time availability of data (Ifenthaler et al., 2014). Lecturers may use rich data for pedagogical decision-making, understand individual performance development of students, identify potential lack of students’ capabilities or the need for curricular improvements (Mattingly et al., 2012). With LA, both students and lecturers can reflect on and improve their communication skills. By capturing, analyzing and visualizing the available information about learning and teaching, lecturers are able to make more reliable predictions about their students’ academic success (Macfadyen and Dawson, 2012; Mah and Ifenthaler, 2018). Furthermore, students at risk can be identified and given support through personalized pedagogical interventions (Lockyer et al., 2013). Successful applications of LA at universities are, for example, Course Signals at Purdue University, aiming to identify students at risk using an approach similar to a traffic light system (green – no risk, yellow – potential risk, red – risky). Students and lecturers can identify needs for action to improve their learning situation. Furthermore, lecturers are able to intervene and help early (Ifenthaler and Schumacher, 2016). At the University of Wollongong, SNAPP (social networks adapting pedagogical practice) is used (Dawson et al., 2011). The main purpose of this system is to increase collaborative learning. For example, conversations in forums are analyzed for investigating the relationship between students and excluded students can be determined. If students have a high learning orientation, they are discussing more frequently in forums about learning and resource sharing. And, students with a high focus on performance are discussing for example about assessments. The interaction delivers data that can be used to get an insight of student’s engagement. At the end of a lecture, the system can also be used for reflection (Sclater et al., 2016). Recent developments include LeAP (LA profiles) which provides students support toward specific learning outcomes in formal learning environments (Schön and Ifenthaler, 2018).

Mobile learning – m-learning – enables learning through personal portable electronic devices across multiple contexts (Delcker et al., 2018; Sampson et al., 2013). Through the use of mobile devices, students have access to learning artifacts more easily. The precondition is the availability of Web-connected devices. In addition, students are independent from locations, time, and they can communicate asynchronously (Lin et al., 2016). Mobile learning supports self-regulated learning (Ifenthaler and Lehmann, 2012) on the one hand, and it is an essential element of blended learning environments on the other hand (Al Saleh and Bhat, 2015). An example for a German mobile application is ARSnova of the THM University of Applied Sciences Gießen. ARS stands for an audience response system. Via this app, students and lecturers can communicate interactively. For example, students can ask questions anonymously during a lecture and the lecturer is able to answer the questions in near real time. Moreover, the communication can take place before or after a lecture. Another function is the evaluation of a lecture directly in the app (ARSnova, 2015).

The app MyLA (My Learning Analytics) of the Cooperative State University Mannheim and University of Mannheim targets to improve learning processes in higher education institutions. MyLA provides ubiquitous communication in form of short messages from students to their lecturer and vice versa. This is especially useful for dual-system courses where students are often away from the campus. Using such data, lecturers can adapt their lectures at university and implement personalized interventions, e.g. adaption of learning materials or a detailed view at topics the students have problems with. Furthermore, the app supports a more personalized and individual way of learning. The students can document their learning motivation through their learning process. The app provides information which enables students to observe their personal progress over time. In summary, MyLA combines LA and m-learning with the extension of personalized learning elements.

The remaining article focuses on the components of the MyLA app prototype and the usability testing of the app. Additionally, the current version of the MyLA app and the MyLA dashboard will be presented.

2. Literature review

Digital media and new technologies have been deployed frequently in the classroom since the 1980s. With the development of wireless internet access, innovative forms of communication and trans-regional collaboration were born. Digitalization and globalization also led to a change in higher education institutions. Particularly, the shift from traditional teaching (i.e. face-to-face teaching) to adaptive and self-regulated learning (SRL) introduced the need for smart learning classrooms. Hence, the benefits for students are a more personalized and independent learning environment (flexibility of time and location) (Hwang, 2014). According to Koper (2014), the following aspects can classify smart learning environments:

  • Physical learning environments enhanced by digital devices.

  • Consideration of learners’ status quo (e.g. culture, context).

  • Digital devices provide additional features for learning, like virtual collaboration or communication, feedback and assessment.

  • The monitoring of learners’ development using analytics results of relevant data.

Particularly interesting in this context and for generating a comprehensive understanding of students learning behavior is the use of advanced data sources. Case studies indicated that learning analytics can:

The first aspect involves that a lecture can adapt the curriculum toward the students’ requirements. As a predictive tool, LA can help to enhance the students’ retention. Furthermore, the learners can use the tracked, analyzed and often visualized data as a self-control mechanism.

A current study investigated three relevant features of LA from the students’ point of view (Schumacher and Ifenthaler, 2018b):

  • Possibility of self-assessment.

  • Recommendations for learning.

  • Timeline for the evaluation of the students’ status quo.

Moreover, privacy is an important aspect when it comes to LA. Students recommend clear transparency and trustfulness of LA applications (Ifenthaler and Tracey, 2016). By providing the users with access to and control over their personal data, confidence can be enhanced (Prinsloo and Slade, 2015). One further aspect is the possibility of making an interpretation of the learners’ information very simple. By visualization of the data, learners, lecturers, as well as other stakeholders can easily identify a good or poor performer (Ebner et al., 2015). Furthermore, individual information can be collected from different sources. While login information and the frequency of particular websites are classically representing quantitative data sources, entries in forums or blog, for example, have to be interpreted as qualitative resources (Ifenthaler and Schumacher, 2016).

Dashboards including vast amounts of data, for example, performance or user data, are a technical solution to support the teaching and learning process. The majority of dashboards used in the context of learning and teaching consider four stages. First, teachers are getting aware of several activities (e.g. the use of learning materials) in the course by using a dashboard. Next, they can reflect concerning their actual teaching process under consideration of the collected data. As a third step, teachers can identify at-risk students or isolated ones more easily. Last but not least, dashboards can be used to become aware of the current impact, and thereby how the information can help to support the students, e.g. through proving opportunities for re-socialization or providing feedback to poor performers (Verbert et al., 2013).

Usability testing for mobile applications considers different factors. For example, Nielsen (1993) mentioned five attributes: learnability, efficiency, memorability, errors and satisfaction. Learnability covers an easy-to-learn approach, efficiency leads to high productivity, memorability helps to remember the system more easily, errors should not occur or can be revoked and satisfaction means that users like the system. But there are more attributes that can be valuable for usability testing. As an example, the MAUEM (mobile application usability evaluation metrics) model has nine attributes. The five of Nielsen and four added attributes: effectiveness, cognitive workload, interruptability and simplicity (Saleh et al., 2017).

A usability example will be introduced of ME2.0 (Mobile Electronic Personality version 2). This is an application that helps users to manage their personalization attributes, e.g. against identity thefts. The methodology covered two studies. In the first study, users (N = 9) read different scenario uses of the application before they were interviewed. In a last step, the participants illustrated their own desired UI and navigation. The second study consisted of an online survey (N = 16) and an evaluation of an illustrated prototype (N = 4) (Oyomno et al., 2013). Another example is EASEL (Education through Application-Supported Experiential Learning). This is a mobile platform, which can be used by instructors to send reflection, as well as content prompts. After a pre-conducted needs analysis, a prototype was built. The participants (N = 14) were divided in faculty members (N = 8) and students (N = 6). Each group had to solve similar (but different) tasks toward a hypothetical scenario. The screen was recorded for analyzing how participants navigated within the application, and the facilitator observed them. All participants were asked to “talk-aloud” while completing the tasks. For documenting comments or gestures, cameras were used to record all sessions. After each task, the participants answered short questionnaires about their experiences. Furthermore, the usability questionnaire was conducted with questions concerning their general use of technology and application-related questions. In addition, all participants were asked to discuss their own experiences with the facilitator (Schnepp and Rogers, 2017).

3. Usability testing

This section focusses on the methodology, as well as on the findings the of MyLA prototype’s usability testing. This is the basis for the actual app and the additional MyLA dashboard, which will be introduced in Section 4.

3.1 App prototype

The Web app prototype of MyLA consists of three different main units (see Figure 1):

  1. My Profile;

  2. My Learning; and

  3. My Progress.

The general page structure consists of the following:

  • Header: with an icon for the side menu, the MyLA logo and an icon for the home button (except on the index page).

  • Main: with contents (e.g. select lists in the profile data or pinboard elements like button and post).

  • Footer: with the linked Cooperative State University Mannheim logo, imprint and privacy, icons linked to: contact, frequently asked questions (FAQ), settings and search function.

My Profile contains two subcategories:

  1. profile data with input options like username or university; and

  2. the trophy center where app users have access to their rewards, i.e. when entering the profile data or participating on a certain questionnaire. This section includes administrative information of every app user.

My Learning contains two subcategories:

  1. the pinboard where students can create posts for their lecturers; and

  2. the survey center where students can respond to regular conducted questionnaires. Within this section of the app, the students can communicate with their lecturers and vice versa. Via pinboard, students can post messages by using tags (e.g. ask questions or point to a problem). This will be only visible for the responsible lecturer on their dashboard interface.

My Progress contains two subcategories:

  1. MyLA data where students can enter personal data like learning motivation or learning effort; and

  2. MyLA stats where the MyLA data will be visualized in charts. This part displays the individual progress in additional (learning) factors.

The design of the app was realized to accommodate the corporate design of the Cooperative State University Mannheim and University of Mannheim. Therefore, one significant color of each education institute had been extracted. As a next step, the colors were combined and supplemented by neutral colors.

According to the approach of LA, the first prototype of the MyLA app was designed to capture user data. Further steps will be collecting the data reports and deriving individual actions for students. Thereby, the main objective of developing personalized and adapted learning environments will be striven. The current app version has been modified and adapted. Additionally, a dashboard for lecturers has been developed.

3.2 Research questions

The usability testing focused on three major research questions:

  • RQ1.How intuitive is the MyLA prototype (design, navigation) for students?

  • RQ2.Is there room for improvement for the development of the MyLA prototype?

  • RQ3.How can the empirical results (quantitative and qualitative) help to optimize the MyLA prototype for its initial implementation?

The questionnaire was conducted to evaluate design, navigation, text elements and used icons of MyLA. Beforehand, the students were shortly introduced to the topic of LA. Afterward, they were able to view the MyLA app either on web browser (Cooperative State University Mannheim) or on mobile browser (University of Mannheim), and afterward responded to an online questionnaire. For the purpose of comparability, none of the participants has used the MyLA Web app before.

3.3 Methodology

3.3.1 Participants.

The usability test was conducted with N = 105 students (N1 = 56 Cooperative State University Mannheim, N2 = 49 University of Mannheim; 51 female, 54 male) in April 2017. The average age of the participants was 23.65 years (SD = 3.72, Min = 19, Max = 35). The majority of the respondents (N = 99) were studying in the field of business administration. More than half of the students were enrolled in a bachelors program (53 per cent) and 47 per cent were studying in a master’s program. In addition, seven students (four female, three male) took part in an additional eye tracking study. On an average, participants reported that they are spending 29 days (M = 28.90, SD = 4.56) per month with apps generally, but only four days per month (M = 3.65, SD = 6.06) with apps for learning.

3.3.2 Design and procedure.

The usability test was divided into two parts using a standardized instrument (see Section 3.3.3): First, the participants made themselves familiar with the MyLA app prototype via web browser or mobile browser. This included the navigation through the app and reviewing the design and the app’s structure. Second, they responded to an online questionnaire which was structured as follows: Socio-demographic information, general usage of mobile devices and technologies, and MyLA-specific questions (open and closed questions). MyLA-specific questions focused on navigation and navigation elements, design and app structure. The main group (N = 98) followed this procedure. A smaller group (N = 7) participated in an eye tracking study (see Section 3.3.4). A significant difference between the participants was the device on which MyLA was tested: One group (Cooperative State University Mannheim) tested on a web browser via a personal computer and another group (University of Mannheim) on a mobile browser via a tablet. To ensure anonymity, various identification numbers had been given to the students.

3.3.3 Usability instrument.

The feedback of the students was committed via an online questionnaire. The question pool within the MyLA-specific part was chosen following the usability testing instrument developed for HIMATT (highly integrated model assessment technology and tools). The instrument has been successfully tested for reliability and validity (Pirnay-Dummer et al., 2010). For the MyLA usability testing, 13 items had been chosen. To give an example, one item was: “I found it easy to navigate through the software”. All questions were answered on a five-point Likert scale ranging from highly agree (5) to highly disagree (1). Figure 2 shows the 13 items used in the usability testing.

3.3.4 Eye tracking.

According to Rayner (2009), eye movements are connected to a participant’s attention, and can therefore contribute to the usability testing of screen-based applications. Eye tracking is a standard methodology to record the eye movement of participants. The data evaluation is conducted with special eye tracking software. A very useful feature is the report function that visualizes the eye movements through heat maps or gaze plots (Kurzhals et al., 2017). Common observations in eye tracking studies include (Ehmke and Wilson, 2007):

  • fixation points: where participants have a long look;

  • first look: where participants look first; and

  • non-looking: elements participants do not pay attention to.

After a short introduction, the students were instructed to solve three tasks by using the app prototype. The difficulty of each task was ascending. Therefore, seven AOI (areas of interests) were defined for the analysis with Tobii Pro Studio software. All participants were recorded with regard to their “first look” and on which MyLA contents they looked more often (“fixation points”). Furthermore, the student’s solution approach was analyzed. The topics of the three tasks can be summarized as follows:

  1. Calling the Cooperative State University Mannheim website as soon as possible.

  2. Selecting and saving their respective university in the subcategory profile data.

  3. Creating a post for a lecturer.

Students from the eye tracking study also participated in the survey-based usability test.

3.4 Results

The presentation of findings is divided into the survey-based and eye tracking usability test. Data analysis was conducted using IBM SPSS 23.0 and Tobii Pro Studio.

3.4.1 Survey-based usability test.

The main part of the questionnaire was to investigate the app’s navigation and design. Figure 2 shows the findings of the 13 items from the survey-based usability test.

The bar charts in Figure 2 are divided into two sections. The chart on left side shows the results concerning the navigation and structure of the MyLA prototype. The second chart on right side highlights the outcomes regarding the design and colors of MyLA. According to the navigation and structure, it is conspicuous that all six average values are constantly high. The highest value was reported for the simplicity of MyLA app prototype with an average of 4.45 (SD = 0.83). Followed by “I found it easy to navigate through the app” (M = 4.36, SD = 0.82) and “The navigation of the app is user-friendly” (M = 4.28, SD = 0.85). Based on the second chart, it is obvious that there were some divergent opinions concerning the design and colors of MyLA. The lowest rated value was the use of color with an average of 3.28 (SD = 1.15). In addition, the participants ranked the design of MyLA as “optically appealing” with 3.36 (SD = 1.19).

3.4.2 Eye tracking usability test.

For the purpose of statistical analysis, either the time of first fixation or the time to first mouse click were calculated. Table I shows the average times the pilot tester (conducted by a research team member) and the participants needed to complete the three tasks. The eye tracking study was implemented using the web browser version of MyLA (on a personal computer).

Table I shows that the majority of the participants (at least five) were able to solve the simulated tasks. For task three, the students needed more time to find the solution in comparison to the first two tasks when compared to the benchmark of the pilot test. A possible explanation may be an unclear description of the task. Some students were not able to find the pinboard on a direct way (via chapter icon “My Learning”). However, all participants completed task three through an alternative solution (via side navigation menu).

As a next step, a heat map analysis was conducted which is a reflection of the screen where the participants viewed longer than other parts (Ehmke and Wilson, 2007), identifying gaze behavior precisely (Duchwoski et al., 2012). Through the accumulation of all single viewpoints of a participant, the fixation points can be highlighted. Figure 3 shows the accumulation of the fixation points recorded by all participants. The viewpoints were predominantly recorded on the left side of MyLA prototype. For solving task one (pictured in the heat map of Figure 3), the participants had to look at the left side to find the Cooperative State University Mannheim logo. Additionally, it has to be considered that the participants saw the app via a desktop browser; hence, the screen width was obviously wider than on a mobile device.

4. MyLA app and MyLA dashboard

In this section, the current applications (MyLA app and MyLA dashboard) will be introduced. The navigation and design were adjusted on basis of the conducted usability testing in spring 2017.

4.1 MyLA app

The app prototype of MyLA has been extended and modified in functions and design. While the app prototype was already assessed as “intuitive” and clear in its navigation, only the color scheme was modified. By realizing app version 1.0, this issue was fixed and a more reduced, neutral and clear color scheme was implemented. The three main categories My Profile, My Learning and My Progress are still the same as in MyLA’s prototype. Besides the style adjustments of MyLA, the backend infrastructure has been developed. And because MyLA app is primary for students, there was an additional need to create an additional surface for lecturers. Therefore, the MyLA dashboard has been developed in line to the MyLA app. The current app version 1.1 (February 2018) has come up with some slide modifications. Students can receive lecture messages, published only to this lecture (compare to Pinboard Entries in the next section). Moreover, the app user can see what is new in the app via a notification bar, added on top of the footer. The number of notifications is also shown near the home button. Furthermore, it is visible in the side menu and on the app’s homepage partitioned by category and subcategory. Additionally, general statistics of the app user can be accessed via the site app statistics. For example, users can see their registration date or the total number of submitted pinboard entries.

Figure 4 shows a screenshot of the MyLA app’s homepage consisting of three links (top-down: My Profile, My Learning, My Learning Progress) to all three categories, mentioned above. On the left side, the side menu is visible and displays all categories and subcategories (top-down: My Profile – Profile data and Trophies, My Learning – Pinboard and Survey, My Learning Progress – MyTracker and MyTracker-Stats, My App Statistics). Furthermore, above the application’s footer, the notification bar is visible (“1 neue Nachricht” means 1 new notification).

4.2 MyLA dashboard

The dashboard version 1.1 has identical functionalities and navigation parts as the application. The dashboard’s homepage contains: Pinboard Entries, Survey Center and LectureTracker. The connection between both applications is managed via profile settings. The instructor has to register a certain lecture and define an access code. This code and the automatically given lecture number are perquisites for joining one particular virtual lecture. Via Pinboard Entries lecturers can receive messages, answer them or mark them as read. Additionally, he or she can filter the entries regarding several tags (e.g. question or problem). Furthermore, instructors are capable to send lecture messages consisting texts and optional links and files. Via the feature Survey Center, lecturers have the possibility to create surveys using existent questions or adding own questions. Because of the LectureTracker, instructors can observe the aggregated MyTracker values of the students. There are three options to filter the data: on a daily, weekly or monthly level. The dashboard is only managed by the lecturer to start an action (e.g. new survey), react on students’ action (e.g. comment pinboard entry) or observe students’ progress (e.g. LectureTracker).

Figure 5 shows a screenshot of the MyLA dashboard’s homepage consisting of three grids with all parts and functionalities, mentioned above. On the left side are the Pinboard Entries (“Pinnwand-Einträge”) and also a collapsible unit to submit a message to the respective lecture (“Kursnachricht erstellen”). In the middle grid, the Survey Center (“Umfrage-Center”) is visible. The two collapsible units display a series of default questions (“Übersicht Fragen-Pool”) and a form to submit own questions with own options (“Eigene Frage zum Pool hinzufügen”). On the right side, instructors can observe the aggregated values of the whole lecture using the LectureTracker (“KursTracker”). All MyTracker data, which will be reported by students, result in the average values. Furthermore, above the dashboard’s footer, the notification bar is visible identically to the MyLA app. The header contains several buttons like: profile, reload and options button (with e.g. imprint and privacy).

5. Discussion and conclusion

Usability testing is a very useful and advantageous method for formative evaluation of the development process (Pirnay-Dummer et al., 2010). The project MyLA can benefit from the valuable input of the potential app user.

With regard to RQ1 (“How intuitive is the MyLA prototype (design, navigation) for students?”), it can be suggested that the prototype is intuitive for the target group. Overall, the results were predominantly positive (all average values were higher than 3); however, some issues were identified for improvement, especially with regard to colors.

In addition, students provided feedback within the scope of open questions. The students evaluated the app’s clarity, simplicity, navigation/structure and features as very positive. Regarding the assessment of the color scheme, the responses were heterogeneous. Critical issues were partly used dark colors, as well as color combinations. Moreover, some students mentioned that the app contained a broad color spectrum. Furthermore, they suggested additional features for future app versions, e.g. a calendar function. With regard to RQ2 (“Is there room for improvement for the development of MyLA prototype?”), it can be summarized that the colors need to be adjusted.

To answer the RQ3 (“How can the empirical results (quantitative and qualitative) help to optimize the MyLA prototype for the main survey?”), the following aspects can be summarized. The involvement of the target group (students) was very important at this early project stage. The reason for that is very simple, because the students are prospective users of MyLA. Therefore, it is inevitable to get them highly involved. The success of a project depends on its acceptance. If the acceptance is high, the potential usage can be high, too. For reaching a large consumption of MyLA, it is necessary to implement the students’ feedback and recommendations. With help of these new insights, future adaptions can be managed. Hence, the findings of the MyLA usability testing provided detailed insights to optimize the app prototype. Some lessons learned of the MyLA usability testing were the following proven statements:

  • The handling of the MyLA Web app prototype is intuitive.

  • The app’s structure is easy to learn.

  • The navigation within the app is clear and user-friendly.

  • The students mostly like the idea of MyLA.

  • The design and colors can be improved, because the opinions deviate fairly high.

With respect to the usability testing and the students’ recommendations, the app was adjusted. Furthermore, an additional application for lecturers was developed (MyLA dashboard). For guaranteeing the functionality of both tools, they were tested with a small sample. One aim of the research project is to focus on the needs of the major target groups (students and lecturers); thus, several workshops were conducted in 2017 and further events are planned for 2018. With respect to the current status of the project, it has to be mentioned that the panel study started successfully at the beginning of 2018. First prefunded results of the study will be available in summer 2018.

MyLA can be used in different learning settings like in lectures, workshops or practical phases. The lecturer can integrate both applications as they fit hihe/sher needs the best. There are no universal approaches, but it is highly recommended to clarify the general setup of using MyLA to the students in an early process stage. The applications can be used for student feedback, communication with the lecturer, (flexible) time management and for performance or lecture evaluations. Via the MyLA dashboard, the lecturer can manage these functionalities and observe the incoming data. Additionally, the identification of further functionalities needed by the students is an evident part of the project.

According to the literature, the research area of LA is still in its infancy in German-speaking countries (Ifenthaler and Drachsler, 2018; Ifenthaler and Schumacher, 2016). Moreover, with the advance of digital technologies, new skills and competencies are required from students and teachers in higher education institutions, as well as their future workplace (Gibson and Ifenthaler, 2017). In addition, the situation of non-traditional academics will be considered within the project. The combination of LA and m-learning seems to be beneficial to contribute to the individual demands of a diverse group of learners. For example, the project team aims to identify patterns of learning behavior and the management of learning tasks, given a high demand of workload at the university and the workplace (Ifenthaler, 2018). In summary, the project can be seen as an important contribution to the topic of LA for higher education in Germany.

Some limitations and challenges of MyLA (and probably other applications) are as follows:

  • Re-designing and implementation of digital tools in curricula.

  • Conviction of lecturers to sustainably use digital offers.

  • Boost of m-learning concepts in higher education.

Some chances of MyLA are as follows:

  • Data collection concerning application usage for research improvement in Germany.

  • Privacy-based approach.

  • Data collection for lecturers to identify certain problems within a lecture.

  • Transfer the Web app to a hybrid app for using device capabilities in future.

As a conclusion of all mentioned points above, there is more research necessary. Some useful research approaches could be: how should a privacy-based application look like in the opinion of students and lecturers, what are overlapping interests of students and lecturers to strengthen and adapt app functions or digital innovations, how can different digital approaches (in one university or cross-university) work together to not “reinvent the wheel”. Furthermore, as already discussed, the additional app features have to be implemented in case of further development. All in all, the project delivers a profound base with many important research questions and approaches for future work.

Figures

Web app prototype MyLA (translated from German; state April, 2017; own figure)

Figure 1.

Web app prototype MyLA (translated from German; state April, 2017; own figure)

Bar charts with the results of the MyLA-specific questions (own figure)

Figure 2.

Bar charts with the results of the MyLA-specific questions (own figure)

Heat map showing the cumulated fixation points of all eye tracking participants (exported using Tobii software)

Figure 3.

Heat map showing the cumulated fixation points of all eye tracking participants (exported using Tobii software)

Screenshot of the MyLA app in German (state: February 2018; own figure)

Figure 4.

Screenshot of the MyLA app in German (state: February 2018; own figure)

Screenshot of the MyLA dashboard in German (state: February 2018; own figure)

Figure 5.

Screenshot of the MyLA dashboard in German (state: February 2018; own figure)

Overview of the eye tracking usability test (all values in seconds)

Tasks Pilot test (N = 1) Participants (N = 7)
01 – Website challenge (click on a logo) 1.91 M = 4.86, SD = 2.46 (solved by 5 of 7)
02 – University selection (in the profile) 10.09 M = 14.55, SD = 9.43 (solved by 5 of 7)
03 – Pinboard post (create a new one) 4.81 M = 10.12, SD = 9.02 (solved by 7 of 7)

References

Adams Becker, S., Cummins, M., Davis, A., Freeman, A., Hall Giesinger, C. and Ananthanarayanan, V. (2017), “NMC horizon report: 2017 higher education edition”, The New Media Consortium, Austin, TX, available at: http://cdn.nmc.org/media/2017-nmc-horizon-report-he-EN.pdf (accessed 05 March 2018).

Al Saleh, S. and Bhat, S.A. (2015), “Mobile learning: a systematic review”, International Journal of Computer Applications, Vol. 114 No. 11, pp. 1-5.

ARSnova (2015), “Feedback: Audience response system für innovative lehre”, Technische Hochschule Mittelhessen (THM), available at: https://arsnova.thm.de/blog/wp-content/uploads/2015/08/ARSnova-Produktprospekt-v5.pdf (accessed 24 May 2017).

Dawson, S., Macfadyen, L., Lockyer, L. and Mazzochi-Jones, D. (2011), “Using social network metrics to assess the effectiveness of broad-based admission practices”, Australasian Journal of Educational Technology, Vol. 27 No. 1, pp. 16-27.

Delcker, J., Honal, A. and Ifenthaler, D. (2018), “Mobile device usage in higher education”, in Sampson, D.G., Ifenthaler, D., Spector, J.M. and Isaias, P. (Eds), Digital Technologies: Sustainable Innovations for Improving Teaching and Learning, Springer, Cham, pp. 45-56.

Duchwoski, A.T., Price, M.M., Meyer, M. and Orero, P. (2012), “Aggregate Gaze Visualization with Real-time Heatmaps”, in Proceedings of the Symposium on Eye Tracking Research and Applications in Santa Barbara, CA, pp. 13-20.

Ebner, M., Taraghi, B., Saranti, A. and Schön, S. (2015), “From the field: seven features of smart learning analytics – lessons learned from four years of research with learning analytics”, eLearning Papers no. 40, pp. 51-55, available at: www.openeducationeuropa.eu/sites/default/files/legacy_files/old/Assessment%20certification%20and%20quality%20assurance%20in%20open%20learning_40_0.pdf (accessed 15 March 2018).

Ehmke, C. and Wilson, S. (2007), “Identifying web usability problems from Eye-Tracking data”, in Proceedings of the 21st British HCI Group Annual Conference on People and Computers, University of Lancaster, United Kingdom, 2007, pp. 119-128, available at: www.bcs.org/upload/pdf/ewic_hc07_lppaper12.pdf (accessed 4 May 2017).

Gibson, D.C. and Ifenthaler, D. (2017), “Preparing the next generation of education researchers for big data in higher education”, in Kei Daniel, B. (Ed.), Big Data and Learning Analytics: Current Theory and Practice in Higher Education, New York, NY: Springer, pp. 29-42, available at: doi:https://doi.org/10.1007/978-3-319-06520-5_4.

Hwang, G.J. (2014), “Definition, framework and research issues of smart learning environments – a context-aware ubiquitous learning perspective”, Smart Learning Environments – a SpringerOpen Journal, available at: https://slejournal.springeropen.com/articles/10.1186/s40561-014-0004-5 (accessed 15 February 2018).

Ifenthaler, D. (2015), “Learning analytics”, in Spector, J. M. (Ed.), The SAGE Encyclopedia of Educational Technology, Thousand Oaks, CA: Sage, Vol. 2, pp. 447-451.

Ifenthaler, D. (2018), “How we learn at the digital workplace”, in Ifenthaler, D. (Ed.), Digital Workplace Learning: Bridging Formal and Informal Learning with Digital Technologies, New York, NY: Springer, pp. 3-8, doi: https://doi.org/10.1007/978-3-319-46215-8_1.

Ifenthaler, D., Adcock, A.B., Erlandson, B.E., Gosper, M., Greiff, S. and Pirnay-Dummer, P. (2014), “Challenges for education in a connected world: digital learning, data rich environments, and Computer-Based assessment – introduction to the inaugural special issue of technology, knowledge and learning”, Technology, Knowledge and Learning, Vol. 19 Nos 1/2, pp. 121-126, doi: 10.1007/s10758-014-9228-2.

Ifenthaler, D. and Drachsler, H. (2018), “Learning analytics”, in Niegemann, H.M. and Weinberger, A. (Eds), Lernen Mit Bildungstechnologien, Heidelberg: Springer, pp. 1-20, doi: https://doi.org/10.1007/978-3-662-54373-3_42-1.

Ifenthaler, D., Gibson, D.C. and Dobozy, E. (2018a), “Informing learning design through analytics: applying network graph analysis”, Australasian Journal of Educational Technology, Vol. 34 No. 2, pp. 117-132.

Ifenthaler, D., Greiff, S. and Gibson, D.C. (2018b), “Making use of data for assessments: harnessing analytics and data science”, in Voogt, J., Knezek, G., Christensen, R. and Lai, K.-W. (Eds), International Handbook of IT in Primary and Secondary Education, (2 ed.), Springer, New York, NY, pp. 1-16.

Ifenthaler, D. and Lehmann, T. (2012), “Preactional self-regulation as a tool for successful problem solving and learning”, Technology, Instruction, Cognition and Learning, Vol. 9 No 1, pp. 97-110.

Ifenthaler, D. and Schumacher, C. (2016), “Learning analytics im hochschulkontext”, WiSt – Wirtschaftswissen-Schaftliches Studium, Vol. 45 No. 4, pp. 172-177.

Ifenthaler, D. and Tracey, M.W. (2016), “Exploring the relationship of ethics and privacy in learning analytics and design: implications for the field of educational technology”, Educational Technology Research and Development, Vol. 64 No. 5, pp. 877-880, doi: 10.1007/s11423-016-9480-3.

Koper, R. (2014), “Conditions for effective smart learning environments”, Smart Learning Environments – a SpringerOpen Journal, available at: https://slejournal.springeropen.com/articles/10.1186/s40561-014-0005-4 (accessed 06 March 2018).

Kurzhals, K., Burch, M., Blascheck, T., Andrienko, G., Andrienko, N. and Weiskopf, D. (2017), “A Task-Based view on the visual analysis of Eye-Tracking data”, in Burch, M., Chuang, L., Fisher, B., Schmidt, A. and Weiskopf, D. (Eds), Eye Tracking and Visualization: Foundations, Techniques, and Applications: ETVIS 2015, Springer, Cham, Switzerland, pp. 3-22, doi: https://doi.org/10.1007/978-3-319-47024-5_1.

Lin, H.H., Wang, Y.S. and Li, C.R. (2016), “Assessing mobile learning systems success”, International Journal of Information and Education Technology, Vol. 6 No. 7, pp. 576-579.

Lockyer, L., Heathcote, E. and Dawson, S. (2013), “Informing pedagogical action: aligning learning analytics with learning design”, American Behavioral Scientist, Vol. 57 No. 10, pp. 1439-1459, doi: 10.1177/0002764213479367.

Macfadyen, L.P. and Dawson, S. (2012), “Numbers are not enough. Why e-Learning analytics failed to inform an institutional strategic plan”, Educational Technology and Society, Vol. 15 No. 3, pp. 149-163.

Mah, D.-K. and Ifenthaler, D. (2018), “Students' perceptions toward academic competencies: the case of German first-year students”, Issues in Educational Research, Vol. 28 No. 1, pp. 120-137.

Mattingly, K.D., Rice, M.C. and Berge, Z.L. (2012), “Learning analytics as a tool for closing the assessment loop in higher education”, Knowledge Management and E-Learning: An International Journal, Vol. 4 No. 3, pp. 236-247.

Nielsen, J. (1993), “Usability Engineering”, Mass.: AP Professional, Cambridge, pp. 26-36.

Oyomno, W., Jäppinen, P., Kerttula, E. and Heikkinen, K. (2013), “Usability study of ME2.0: user interface design for mobile context enhanced personalisation software”, Personal and Ubiquitous Computing, Vol. 17 No. 2, pp. 305-319, doi: 10.1007/s00779-011-0495-9.

Pirnay-Dummer, P., Ifenthaler, D. and Spector, J.M. (2010), “Highly integrated model assessment technology and tools”, Educational Technology Research and Development, Vol. 58 No. 1, pp. 3-18, doi: 10.1007/s11423-009-9119-8.

Prinsloo, P. and Slade, S. (2015), “Student privacy self-management: implications for learning analytics”, in Proceedings of the Fifth International Conference on Learning Analytics And Knowledge, Poughkeepsie, New York, NY, USA, pp. 83-92, doi: https://doi.org/10.1145/2723576.2723585.

Rayner, K. (2009), “Eye movements and attention in reading, scene perception, and visual search”, The Quarterly Journal of Experimental Psychology, Vol. 62 No. 8, pp. 1457-1506.

Saleh, A., Ismail, R. and Fabil, N. (2017), “Evaluating usability for mobile application: a MAUEM approach”, pp. 71-77, In Proceedings of the 2017 International Conference on Software and e-Business, Hong Kong, 2017, doi: 10.1145/3178212.3178232.

Sampson, D.G., Isaias, P., Ifenthaler, D. and Spector, J.M. (Eds) (2013), Ubiquitous and Mobile Learning in the Digital Age, Springer, New York, NY.

Schnepp, J. and Rogers, C. (2017), “Evaluating the acceptability and usability of EASEL: a mobile application that supports guided reflection for experiential learning activities”, Journal of Information Technology Education: Innovations in Practice, Vol. 16, pp. 195-214.

Schön, D. and Ifenthaler, D. (2018), “Prompting in pseudonymised learning analytics – implementing learner centric prompts in legacy systems with high privacy requirements”, paper presented at the International Conference on Computer Supported Education, Funchal, Madeira, Portugal, (accessed 15 March 2018).

Schumacher, C. and Ifenthaler, D. (2018a), “Why learning analytics need to care for motivational dispositions of students”, paper presented at the AERA Annual Meeting, New York, NY, 13-04-2018.

Schumacher, C. and Ifenthaler, D. (2018b), “Features students really expect from learning analytics”, Computers in Human Behavior, Vol. 78, pp. 397-407, doi: 10.1016/j.chb.2017.06.030.

Sclater, N., Peasgood, A. and Mullan, J. (2016), “Learning Analytics in Higher Education: A review of UK and international practice: Full report”, Jisc.

Statista/Bitkom (2017), “Anteil der Smartphone-Nutzer in deutschland nach altersgruppe im jahr 2016”, available at: https://de.statista.com/statistik/daten/studie/459963/umfrage/anteil-der-smartphone-nutzer-in-deutschland-nach-altersgruppe/ (accessed 28 April 2017).

Verbert, K., Duval, E., Klerkx, J., Govaerts, S. and Santos, J.L. (2013), “Learning analytics dashboard applications”, American Behavioral Scientist, Vol. 57 No. 10, pp. 1500-1509.

Acknowledgements

The authors acknowledge the financial support by the Baden-Württemberg Ministry of Science, Research and the Arts (MWK, project reference MobileLA).

Corresponding author

Matthias Kuhnel can be contacted at: kuhnel@bwl.uni-mannheim.de