A case study on narrative structures in instructional MOOC designs

Elke Höfler (University of Graz, Graz, Austria)
Claudia Zimmermann (University of Graz, Graz, Austria)
Martin Ebner (Department of Educational Technology, Graz University of Technology, Graz, Austria)

Journal of Research in Innovative Teaching & Learning

ISSN: 2397-7604

Article publication date: 3 April 2017

2923

Abstract

Purpose

The purpose of this paper is to share the lessons learned in implementing specific design patterns within the “Dr Internet” massive open online course (MOOC).

Design/methodology/approach

MOOCs are boasting considerable participant numbers, but also suffer from declining participant activity and low completion rates. Learning analytics results from earlier xMOOCs indicate that this might be alleviated by certain instructional design patterns – critical aspects include shorter course duration, narrative structures with suspense peaks, and a course schedule that is diversified and stimulating. To evaluate their impact on retention, the authors have tried to implement these patterns in the design of the “Dr Internet” MOOC.

Findings

Statistical results from the first run of the case study MOOC do not indicate any strong influences of these design patterns on the retention rate.

Research limitations/implications

With inconclusive statistical results from this case study, more research with higher participant numbers is needed to gain insight on the effectiveness of these design patterns in MOOCs. When interpreting retention outcomes, other influencing factors (course content, pacing, timing, etc.) need to be taken into account.

Originality/value

This publication reports about a case study MOOC and gives practical hints for further research.

Keywords

Citation

Höfler, E., Zimmermann, C. and Ebner, M. (2017), "A case study on narrative structures in instructional MOOC designs", Journal of Research in Innovative Teaching & Learning, Vol. 10 No. 1, pp. 48-62. https://doi.org/10.1108/JRIT-09-2016-0005

Publisher

:

Emerald Publishing Limited

Copyright © 2017, Elke Höfler, Claudia Zimmerman, Martin Ebner

License

Published in the Journal of Research in Innovative Teaching & Learning. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode


Introduction

With its origins in Northern America, massive open online courses (MOOCs) have conquered not only Europe but educational systems worldwide. The phenomenon can indeed be called global, as Anant Agarwal (2016), the CEO of edX (www.edx.org; one of the most popular American MOOC platforms) has done. He points out “that MOOCs have […] demonstrated potential benefit as a catalyst for change within universities and all over the world” (Agarwal, 2016). Although completion rates are typically fairly low compared to the number of enrolled participants, Agarwal (2016) continues, “MOOC platforms have helped entire countries build their labor forces and create conduits for dramatic social change.” MOOCs can help in making knowledge available for everybody without limitations in time and space, at least theoretically. However, the so-called democratizing process of education that MOOCs were supposed to trigger did not take place in the way experts predicted it almost ten years ago (Dillahunt et al., 2014; Hansen and Reich, 2015), yet Laura Pappano (2012) still declared “the year of the MOOC.”

The MOOC’s dropout problem

In the time that has passed between these first expectations regarding MOOCs and their current standing, scientific research has focused on different aspects within this area. In addition, MOOCs have undergone classification, several subtypes have been identified and still more seem to be forthcoming. Researchers talk about different “MOOC Derivatives” (Hollands and Tirthali, 2014) for slightly modified types of MOOCs, like the personal open online course, the mini-MOOC, or the small private online course, to name just a few. The original dichotomy of xMOOC (based on traditional course structures) and cMOOC (focused on interaction and connectivist principles) was forced to expand. Still, different instructional types and design approaches have led to multiple occurrences of the same phenomenon, like high dropout and hence low completion rates (Colman, 2013; Fidalgo-Blanco et al., 2014; Khalil and Ebner, 2014).

As a result, there is a growing body of recommendations from experts and MOOC providers (usually based on learning analytics, see Veletsianos et al. (2016), results from earlier xMOOCs, specific empirical research as well as literature reviews, such as Bonk et al., in press) with regard to strategies against the decrease in participant activity (Jasnani, 2013; Khalil and Ebner, 2013a, b).

If we emanate from “the typical completion rate of below 10%, approximately 7.5%” (Jasnani, 2013, p. 6) or even less, namely, “from around 3% to 15% of all enrollees” (Hollands and Tirthali, 2014, p. 42), it is not surprising that the investigation of possible reasons for this phenomenon is a pressing concern (Colman, 2013; Khalil and Ebner, 2013a, b). In his online survey, Dan Colman (2013) identified several reasons why participants dropped out instead of finishing a MOOC they had registered for. There are various criteria tightly linked to unfilled expectations of the learner; for example (all quotations cited from Colman, 2013):

  • The time component: “Takes Too Much Time.”

  • The knowledge component: either the course “Assumes Too Much Knowledge” or it was “Too Basic, Not Really at the Level of Stanford, Oxford and MIT.”

  • The instructional (design) component: there might be “Lecture Fatigue” due to the fact that “MOOCs often rely on formal video lectures.” Participants think “MOOCs would be better served if they relied more heavily on interactive forms of pedagogy.” This aspect leads to the second instructional argument, namely, “Poor Course Design” and the wrong decision of tool usage, namely, “Clunky Community/Communication Tools.”

  • The financial component: “Surprised by Hidden Costs.”

Besides the learner’s personal expectations, there are criteria that focus on the peer group, its behavior, and the kind of socializing within the MOOC:

  • The social component: “Bad Peer Review & Trolls.”

    Finally, there is the participants’ attitude toward the MOOC itself that can be seen as an important criterion whether to attend and complete a MOOC or not:

  • The personal component: it was not the primary goal to do a MOOC or to finish it:

“You’re Just Shopping Around” as only parts of it are interesting and important: “You’re There to Learn, Not for the Credential at the End.”

As can be seen, these reasons are pretty heterogeneous and range from very personal, individual components to more general course-related components and the peer group in a wider sense. These components can be seen as important nodes in the sense of a connectivist approach, the “Learning Theory for the Digital Age” as George Siemens defined back in 2004. Crucial principles as described by Siemens (2004) are that “Learning and knowledge rests in diversity of opinions,” as well as “Learning is a process of connecting specialized nodes or information sources.” There are two important core skills, namely, “Nurturing and maintaining connections is needed to facilitate continual learning” and the “Ability to see connections between fields, ideas, and concepts” (Siemens, 2004). The learners themselves get to decide what and how they want to learn, and in what ways they want to deepen their learning experience in specific fields of knowledge.

With these core skills in mind, Hanan Khalil and Martin Ebner (2013a, b) looked at the different possibilities of interaction within MOOCs and the influence these interactions have on the learner’s motivation. They found out that interaction is a crucial aspect in MOOCs when it comes to the learner’s satisfaction, which in turn is a main decision-making aid when deciding on whether to continue or drop out from a MOOC. In their studies, the authors concluded that there were at least three relevant forms of interaction: among learners; between learners and teachers; and between learners and learning resources, i.e. the content. These three actors form the so-called instructional triangle, as shown in Figure 1.

Kelli Nipper and Paola Sztajn (2008) adopted the instructional triangle for mathematics and added the interactions among students to the interdependence between students and teacher, and content, respectively. Students in this case are the learners or “participants” in a more conventional sense, the teacher is the “organizer” and mathematics is the “content” (Nipper and Sztajn, 2008, p. 336). Regarding the contexts, there are several aspects that warrant consideration, for example, the course design (in contrast to the course content), the financial aspects (as mentioned above), and more general institutional aspects. With the triangle in mind, instructional concepts have been aiming at creating a course design that satisfies not only these three stakeholders, but also adapts to the specific MOOC-related context in a way that enhances and the learner’s experiences and his motivation to continue the course.

Instructional designs for MOOCs

Learning contexts also play a role in traditional classroom or conventional online teaching, but it would seem that they are even more important when it comes to designing MOOCs. In their pedagogical guideline for MOOC practitioners, Elke Lackner et al. (2014) listed six categories that have to be taken into account during the different phases of MOOC design and development. These phases are based on ADDIE (Kopp and Lackner, 2014), the most commonly known instructional design framework focused on core requirements, structure, participant requirements, assignments, media design, communication, and resources (Branson et al., 1975). The teacher’s role is redefined and/or enriched to become a teacher-facilitator-moderator, as well as the choice of the content, the course language, and the participants’ role. The course creator, i.e. teacher, has to decide whether they choose a rather specialized and narrow topic that usually only allows them to address a small target group of “experts,” or a topic that might be of interest for a broad and diverse audience. The same holds true with regard to language: if they decide to do a course in English, the target group is potentially larger than that with a national language such as Italian or German. However, if the teachers want to address a non-academically inclined target group in a specific region, it might be important to choose the regionally spoken language.

The audience, i.e. the participants, is the most crucial player when it comes to course concepts. The teacher can neither predict the number of participants that attend a MOOC nor their social, economic or geographical background, and when thinking of a specific target group during the conceptual phase the teacher cannot be sure that the attending audience will match the fictive target. Additionally, as Colman (2013) showed, a bad peer group or trolls are indeed reasons to abandon a MOOC; hence, the social context of the participants, related to a main claim of connectivist learning settings, must not be neglected. Kopp and Lackner (2014) come to the conclusion that MOOCs need a special instructional design that accounts for an unpredictable, heterogeneous audience and their specific expectations and behaviors, in order to foster participation and retention. Last but not least, learners exhibit different types of motivation to attend a course, and they need specific competences, such as the ability to learn in a self-directed way, in order to make the most of a MOOC (Bonk et al., 2015; Song and Bonk, 2016). This learning process may take place inside the MOOC or within a wider context outside the course, e.g. social networks (Veletsianos et al., 2015).

Design patterns for MOOCs

As traditional instructional design strategies seem to struggle with the challenges created by MOOCs, alternative instructional approaches should be considered. One promising approach that has already received a lot of attention is the application of specific instructional design patterns. They have a tremendous influence on the learner’s motivation, which in turn is crucial for the instigation of learning processes and continuation (Lackner et al., 2015). In contrast to traditional instructional guidelines, “patterns support a wider variety of application scenarios” (Bauer and Baumgartner, 2012), since not only the subject itself but also the context has to be considered. As Reinhard Bauer and Peter Baumgartner (2012) sum up: “Applied to a concrete design process this means that it is not possible to design an isolated ‘thing’ […] ignoring the world around this artifact […] and within this artifact […]. The goal is a holistic approach” in course design that does not only integrate immanent factors, as proposed by traditional instructional design models like ADDIE (Branson et al., 1975), but also covers experiences and is based on good practices as well as existing designs and concepts. Hence, an effective instructional design pattern combines “a clear articulation of a design problem and a design solution […] offering a rationale which bridges between pedagogical philosophy, research-based evidence and experiential knowledge of design” (Goodyear, 2005, p. 92). If a design problem or challenge presents itself, specific design patterns may help in finding an adequate solution.

In their paper MOOCs as granular systems: design patterns to foster participant activity Elke Lackner et al. (2015) formulated a design proposal for MOOCs that was based on learning analytics results from preceding xMOOCs and an extensive literature review. This main theme of this design proposal can be summed up under the term granularity. From this rather generic design aspect, three major design patterns were deduced.

Four-week MOOCs

Research and experience have shown that most MOOCs last about four to eight or sometimes even 12 weeks (Jasnani, 2013). However, there is evidence to suggest that a “concept of ‘modularity’” is advisable, and that “shorter courses are both easier to create and to complete” (Hollands and Tirthali, 2014, p. 92), because “granular courses are more digestible” (Jasnani, 2013, p. 14). Since the most dominant point of dropout seems to be week 4, the design pattern suggests the construction of four-week courses, which accommodates one facet of the above mentioned time constraints as a reason for dropout (Colman, 2013). Furthermore, it is easier for the participants to schedule learning frames within a shorter period of time, i.e. four weeks instead of eight or even 12 weeks.

Granular certificates

As orientations and motivations toward learning and learning systems are always changing, new certification processes should be considered. One step might be the integration of badges, which have been shown to help keep up learners’ motivation (Santos et al., 2013; Wüster and Ebner, 2016). As Colman (2013) pointed out, participants of MOOCs do not always wish to finish the MOOC as a whole – instead they are interested in particular segments, weeks or modules (see “personal component” above). A certificate or a statement of accomplishment is typically issued at the end of a MOOC, once a participant has gone through a determined, complex syllabus. Those who just attend specific modules or select topics do not have any visible acknowledgment of their learning process, which seems to be a fairly typical issue in informal learning processes in general. Badges for portions of the course could help solve this design problem (Wüster and Ebner, 2016).

Suspense peak narratives

Storytelling and a fascination for stories are a deeply rooted anthropologic trait “Wir Menschen sind erzählende Wesen” (Spath and Foerg, 2005) – “We humans are narrating beings.” Human beings need stories, they have an interest in them, they like to narrate and listen to stories. This anthropologic desire can be used for marketing issues (Spath and Foerg, 2005), in journalism (Flath, 2012) as well as for effective learning settings (Koening and Zorn, 2002; Salpeter, 2005). Ever since Medieval Age storytelling, people have used micro-learning units and specific narrative structures to animate the audience to follow narration (Bakker, 1993). Nowadays, some aspects of this literary phenomenon are often employed in modern television formats or serialized novels, for example, in the form of the so-called cliffhanger: “A Cliffhanger plot device ensures readers will buy the next installment in order to read and find out what happens” (LiteraryDevices Editors, 2016). Within MOOCs, this device could form a design pattern when tasks are set like riddles, meaning that a question will only be answered in the following week, combined with encouragement for the participants to discuss possible solutions in a forum with their peers, thus also increasing interaction (a connectivist principle). Another possible manifestation of this design pattern concerns the video resources often used in MOOCs. A longer video can be split up in two shorter video sequences with the cut made in a suspense-packed, tragic or highly informative moment. Ideally, the videos themselves would be telling a story, consisting of role plays or a dialogic and hence dialectic structure; as a consequence, the suspense peak pattern can be realized more intensely.

This suspense peak or cliffhanger design element is one out of seven “elements of digital storytelling” (University of Houston, 2016), identified by the Center for Digital Storytelling (CDS), namely, the “Dramatic Question” which, according to the CDS, opens a question or a problem that is solved only in the end of a story in order to create the viewer’s necessity to stay tuned. These elements can be seen as traditional narrative elements that derive from a long literary tradition as categorized by Gérard Genette (1972) in the early 1970s in his book Figures III. Narratology according to Genette deals with different elements of text: the narrator’s voice, the duration, frequency, and mode of narration, the latter comprising the narrator’s perspective and distance (i.e. its focalization). The six remaining elements are the following.

“Point of View” means the author’s or narrator’s perspective as described by Genette in mode and voice. Furthermore, the author’s or narrator’s basic attitudes play an important role within this element as they give the audience a hint on how to read a story. “Emotional Content” can be described as content that affects the audience and establishes ties between the different stakeholders, similar to the ties within the instructional triangle (see Figure 1). This element is accompanied by a fourth one, called the “Gift of Your Voice,” i.e. the means to achieve a personal connection with the story and that helps the audience to understand its course and its context. The fifth and sixth element, “Economy” and “Pacing,” coincide with Genette’s duration and frequency. “Economy” means the tightrope walk to give as much information as necessary but to not too much. The audience should not be overtaxed yet well informed by the content. “Pacing” means that the audience has the possibility to consume content in their own pace, according to their own needs. The story’s progression should neither be too fast nor too slow, neither overwhelming nor boring. The last of the seven design elements of digital storytelling, the “Power of the Soundtrack,” means the power of audio elements such as sounds and music, which enrich a story, accompany or contrast the narration, create a specific atmosphere and guide the audience. Music and sounds can play an independent role within a story or support the narration (think, e.g. of the music’s role in horror stories).

Our main question concerning the case study is if those described instructional patterns have an influence on learners’ behaviors with regard to their retention rate.

The Dr Internet MOOC – a practical example

The existing body of research on MOOC retention highlights many areas that could prove fruitful for improvement efforts. For example, findings with regard to dropout motivation can and should inform didactical models on the one hand and practical aspects of MOOC design on the other. In this paper, we would like to focus on the efficacy of narrative design patterns, and in order to put these theoretical insights into practice and to evaluate their effectiveness in improving MOOC retention, they were implemented in the best possible way during the conceptualization phase, design, and development of a fairly unique xMOOC called “Dr Internet.” This course was the centerpiece of a comprehensive research project, which is the main reason why it was selected for additional testing of some of the described design patterns: there was an interdisciplinary team of researchers and practitioners involved in designing and implementing the MOOC, more extensive data collection, and an attractive topic that was expected to procure large numbers of participants that would help to study activity patterns.

Background

General practitioners as well as specialized physicians have noticed an increasing tendency among their patients to visit the doctor’s office with previously acquired medical information, obtained from online sources like popular websites, patients’ forums, etc. The extent of the medical knowledge available online can be quite impressive, but has often been found to be inconsistent and difficult to evaluate (Benigeri and Pluye, 2003). Consulting the internet on medical issues bears some obvious risks: under- or overestimation of the severity of physical symptoms, insecurity about doctors’ competence and a subsequent lack of trust, or downright substitution of professional medical treatment for advice from questionable sources, to name just a few. All of these risks might impair the health outcomes of patients (Robertson et al., 2014). However, there is also a tremendous potential to democratize the access to medical knowledge (and its understanding), which has previously been a highly restricted privilege, and to boost health literacy among the general population and especially within some hard-to-reach social groups (Brodie et al., 2000).

This phenomenon is the focus of an interdisciplinary research project on which three Austrian universities cooperate: the Karl Franzens University, Graz University of Technology, and the Medical University (all based in Graz, Styria). The central research question is about whether the internet helps or hurts in the endeavor to increase health literacy among the general population and how these developments affect the physician-patient-relationship. The project was designed to engage the topic on various levels; there is a sociological branch of research that is concerned with empirical studies (interviews with general practitioners and a questionnaire-based survey among their patients), a philosophical arm that focuses on ethical issues in this regard, and various smaller segments.

The central component of this research project is the Dr Internet MOOC, which is accessible at the first and so far only Austrian MOOC platform called “iMooX” (www.imoox.at), hosted by the Karl Franzens University and the Graz University of Technology (Ebner et al., 2015). The videos, all specifically created for this course, are licensed under a Creative Commons License, so anybody is free to access and use them as long as there are no commercial interests involved, as so-called open educational resources (Ebner et al., 2016). The materials will continue to stay available even after the initial MOOC and the research project have concluded.

Narrative MOOC design

The task presented to the learners was to assess and diagnose several medical case studies that were visualized in the form of short videos, in which a patient would display or describe physical symptoms of illness. After watching the video content, the participants were instructed to use the internet in order to research possible diagnoses for the patient in the video, and then take a quiz on the case study. The primary didactic aim of the course was to train competences rather than just increasing knowledge. It was advertised as an opportunity to develop and compare one’s own skills in internet-based diagnosing in a safe environment, thereby raising awareness for a controversial topic and instigating critical thinking processes when it comes to evaluating information and opinions (Zimmermann et al., 2017).

The MOOC was created with two main objectives in mind: to gather data that would complement the other research efforts (Zimmermann et al., 2016), and to raise awareness for a critical but prolific approach to online searches for health information. For both purposes, it was necessary to attract a sizeable number of participants and to design an attractive MOOC that would keep them engaged in order to avoid high dropout rates. We thus tried to implement several key design elements as described by the CDS that have been identified as potentially helpful in fostering MOOC retention (Lackner et al., 2015; Sadik, 2008) and can be labeled as narrative structures.

Several of the design patterns that have been discussed above were part of the conceptual framework during the design and development phase of the Dr Internet MOOC and they were implemented in quite a few essential aspects, six of whom should be scrutinized in more detail: content adaptation to the target group, a special interactive quiz format, a course structure based on the “storytelling approach” that includes serial videos to create suspense peaks and a framing narrative incorporating a narrator, a shorter course duration and a video schedule that offers stimulation and variety, yet is easy to follow.

Choice of the content

The choice and presentation of the content can be crucial when it comes to addressing the intended target group (Lackner et al., 2014). The Dr Internet MOOC and its objective to help people in navigating a changing understanding of the relationship between doctors and patients as well as managing the wealth of online information that could be relevant for their physical well-being was assumed to qualify as so-called emotional content for a large group of people. Thus, it was expected to attract a large and heterogeneous group of participants. To meet their predictably heterogeneous needs and expectations the course design was adapted and the focus was set on a narrative structure that should be activating and easy-to-understand though interesting-to-follow.

Special interactive quiz format

MOOCs typically use multiple choice questions to test the participants’ knowledge gain, but this format was not applicable when judging medical case studies. Since there are no definitive “right” and “wrong” answers when it comes to diagnosing, and standard testing would not really meet the requirements of the above mentioned emotional content, we designed an interactive quiz format that is closer to a poll than a traditional knowledge test. The participants were given a selection of eight diagnoses for each case study, and were asked to rate their likelihood on a four-part scale starting with “unlikely” as the weakest and ending with “very likely” as the strongest category. While there was no feedback on whether or not their estimations were correct, participants did have the opportunity to compare their results to the collective ratings of a group of trained physicians who had been surveyed before the start of the MOOC. Additionally, there was also the option of comparing one’s own choices to those of other MOOC participants who had already taken the quiz. These averaged quiz results were updated instantly, and with more people casting their votes, the group result was constantly changing and could look quite different at various points in time, therefore providing an incentive for the participants to come back later and check it more than once.

Serial videos

Leaning on the narrative approach, we constructed two more extensive case studies where the story of one patient was told in two videos instead of one. As a result, these serial videos were released over the span of two weeks. The supposed “cliffhanger” effect, or the dramatic question as it has been termed by the CDS, was used to provide an incentive to stay engaged in the MOOC and to come back to this case study in the following week. To similar effect, all of the medical cases had a corresponding “resolution video” that was also released in the subsequent week, and in which the general practitioner who created the case studies explains which one of the proposed diagnoses he believes to be the most likely one and why.

Narrator

This general practitioner acts as narrator and a golden thread guiding through the different video episodes, i.e. the frame of the stories narrated within the MOOC. Against the tradition introduced by Boccaccio in his fourteenth century Decameron, a collection of 100 novellas, there is no macro story. The frame does not narrate a fictive story but acts as the frame for several fictive stories. According to Genette’s (1972) typology, the narrator’s voice is hetero-diegetic and extra-diegetic, as he is not part of the novellas, i.e. the videos. He is, though, a person the participants can trust in, since he is an expert in this field, as opposed to the concept of an unreliable narrator, which is the main component of the cliffhanger structure. His appearance on a weekly basis serves as the frame that organizes the process of content reception. Regarding focalization, the narrator is non-focalized, as he knows, tells, and considers the thoughts and stories of each protagonist in his personal analysis, i.e. the above mentioned gift of your voice. The audience thus gets more familiar with the case studies, the general practitioner’s experience, and opinion.

Duration

The standard MOOC duration of eight weeks was reduced to six weeks. Previous research on MOOCs indicated that there is a typical decline in activity and increase in dropouts around the fourth week. While this would imply an ideal course duration of four weeks, we had to take other organizational factors into account and arrived at the compromise of six weeks duration. Each week’s workload was estimated with three hours, which included watching the video case study, doing online research to master the quiz and engaging in forum discussion. Economy, the digital storytelling device named by the CDS, guided the selection of the resources and design of the activities, and the three-hour-model was calculated on a generous basis. With regard to the pacing device, there were no fixed appointments or dates within these six weeks, except for the fact that the course was timed on a weekly basis and started Monday mornings with a new chapter and the corresponding materials.

Video schedule

Due to the six weeks duration of the course, it was decided that six case studies would be an adequate number. The devices of economy and pacing led the conceptual considerations, and since it would have been very predictable and not particularly engaging if the presentation followed a simple model of one case per week, we constructed a more diversified video schedule (see Table I). The idea was to provide a stimulating variety of videos on offer that would neither bore nor overwhelm the participants. The videos themselves were rather short, with an average duration of about five minutes. The relation between discourse time and narrative time can be described as focusing on a short discourse time, and doing so without pro- and analeptic references, i.e. flash-backs or flash-forwards. Obviously, a chronological narration that is reduced to one main narrative thread is easier to follow, but more likely to arouse boredom; which was considered to be only a small risk with videos of such short length.

Results from the first run of the Dr Internet MOOC

Due to the fact that the MOOC was only one part of a bigger research agenda, participants were required to fill in a mandatory questionnaire before entering the MOOC for the first time. This questionnaire mainly included questions on health-related matters and previous experiences when looking for health information on the internet, but it also provided a set of basic sociodemographic variables[1]. Overall, the MOOC did not attract a particularly large crowd of users during its first run (October 27-December 6, 2015). Out of 370 registered participants, 206 did in fact enter the MOOC and completed the questionnaire (56 percent), thus qualifying as active learners. The majority of these were female (62 percent) and the average age was 39 years (minimum 15 years, maximum 75). Singles constituted the dominant civil status (58 percent), followed by married (32 percent), divorced (8 percent), and widowed (2 percent) users. With regard to educational level, we see a familiar picture that has been reported from many other MOOCs (Dillahunt et al., 2014; Hansen and Reich, 2015; Neuböck et al., 2015): well-educated participants are heavily overrepresented, with almost 90 percent having completed either high school (33 percent) or a university degree (56 percent).

For an evaluation of the implemented design elements, we obviously have to look at dropout and activity rates throughout the course. Anybody who either took a quiz, read or wrote something in the forum, or watched a video in any given week was classified as an “active participant.” Again, we encounter some fairly typical results: 124 active participants started out in the first week, and 52 still remained in the last week of the course. Figure 2 gives an overview of the decline in the numbers of active participants: as we can see, the biggest drop in participant numbers did not occur in week 4, but between weeks 2 and 3, where there was a loss of almost 40 percent (from 139 down to 84 users). After that, the numbers are stabilizing, and the decline is almost linear. When comparing the small group still active in week 6 (n=52) with the big group that filled in the questionnaire in the beginning (n=206), it can be noted that most sociodemographic variables change very little (gender proportions are now almost equal, single participants drop out at a higher rate than married ones, and the proportion of university-educated is only slightly higher than of those who completed high school). Overall, it can be said that the dropout rates seem to have a weak relationship with sociodemographic characteristics.

The interpretation of these results regarding the effectiveness of the implemented design-based elements is not straightforward. On the one hand, weeks 2 and 3 featured the first two-week video, which coincides with the highest loss in user numbers over the six weeks course duration (see Figure 2). On the other hand, the second two-week video was released during weeks 4 and 5, where we see only a slight drop in numbers. After the initial peak in week 2 and the following week’s loss, the slope of the curve is gradual rather than steep, so there is likely no single effect in the design that caused direct and noticeable dropouts among all participants. When looking only at those participants who completed any quiz in each week, there is a much less pronounced drop in numbers after week 2, and an actual increase between weeks 4 and 5. This could indicate that there is a subgroup of participants that is more interested in the video and quiz features of the MOOC, and that to them the storytelling approach might have been more attractive.

The overall decline is substantial, yet compared with the drastically low completion rates that are usually associated with MOOCs (Hollands and Tirthali, 2014; Jasnani, 2013), the result is quite adequate. Depending on whether the completion rate is calculated as a percentage of registered or active users, the Dr Internet MOOC achieves 8 and 14 percent, respectively. Still, these results do not mark a significant improvement in retention as had been expected from the theoretical assumptions discussed above. In summary, this MOOC did not produce any solid evidence to indicate that the implemented design-based elements were in fact making a difference.

Discussion and conclusion

While the results from the Dr Internet MOOC do not provide any support for the effectiveness of the design patterns in questions, there is also no clear indication that they do not work. The attendance numbers fell short of expectations, so further research with larger sample sizes is needed to produce more data on the subject. While it is hard to discern the reason for the rather small turnout, several factors can be assumed to play a role: the timing of the course dates at the beginning of the school semester, which is generally a good time, but also means that this MOOC was competing for participants with a few other courses on the same platform, the channels used for advertising, in this case mainstream media rather than targeted promotion for an internet-affine audience, and the (academically speaking) low-threshold topic, since the typical MOOC users are disproportionately well educated (Dillahunt et al., 2014). It might also be possible that the compulsory questionnaire at the beginning of the course could have kept some registered users from active participation, although there is no evidence for this assumption when looking at the data – 56 percent of those enrolled qualify as active learners, a fairly average proportion (Koller et al., 2013).

The lack of retention effects could be due to a variety of reasons, for example, the specific way we implemented those principles – the screenplay for the videos might not have delivered the right kind of cliffhanger, and the suspense peaks might not have been perceived as such. The narrator’s framing role could have been developed in a more pronounced and less subtle way – we did not include an introductory video in which the general practitioner introduced himself and his role within the course structure. It might also have been more effective to let the general practitioner activate the forum discussions to strengthen his role as a guide. Additionally, while the MOOC’s topic was fairly attractive in the eyes of the press, it was also unconventional in the sense that it did not convey a lot of factual knowledge, and thus the results may not be representative of more traditional MOOCs. In fact, it is highly plausible that the kind of knowledge conveyed in MOOCs (e.g. competence building vs knowledge gain) determines some of their specific characteristics (like materials and activities offered) and thus also affect the effectiveness of some design patterns.

Based on the data from this course and experiences with previous MOOCs, we have identified another potential design problem in line with the pacing device that could have a significant influence on participants’ dropout: the scheduling of weekly modules. It is tempting to think that the waiting time for either a case resolution or more videos of the same interesting topic is something that persuades participants to log in next week when new content is available. However, this model similar to the suspense peak approach might not work for everyone, and there is evidence that some people actually lose interest when they have to wait for access to the materials they want to see. They might forget about it in the meantime, they are not interested in making the MOOC part of their weekly routine, or they could be unable to take time out of their busy schedules on a regular weekly basis. Data from the Dr Internet MOOC shows that there is only a very small group of users who log in weekly and participate in the available features as they are released. The overwhelming majority has fewer logins, but many of them catch up on material that is new to them.

This phenomenon has inspired some debate about self-paced as opposed to session-based MOOCs. Both approaches come with benefits and disadvantages: self-paced learning without fixed start and end dates cannot provide the same kind of community of many participants learning the same things at the same time, and there are no deadlines, which to some can be a powerful motivation to complete course tasks (Shah, 2015). However, it would seem that the self-paced concept is gaining momentum; a survey conducted among European institutions of higher education revealed that a subsample of MOOC providers attributed higher relevance to the freedom of learning at one’s own pace than to fixed start and end dates (Jansen and Schuwer, 2015).

There are several measures that could help to address this particular variation in learners’ preferences. On the one hand, there is the option of structural changes to the weekly schedules of MOOCs in the direction of self-paced learning, and a higher degree of learner empowerment in general (Guàrdia et al., 2013). On the other hand, the session-based concept could benefit from e-mail reminders for the participants to inform them about new content and to encourage them to return to the course. Even though learner’s structural preferences might vary among the population of MOOC participants, it is reasonable to assume that a stronger focus on these preferences is in any case a promising strategy for future research on MOOC retention.

While course design is among the most important factors for retention in MOOCs, it is far from being the only one. There is always a certain fraction of participants who enroll for MOOCs because of the interesting topic or one specific aspect they are interested in. Earning a credential, an acknowledgment or a statement of accomplishment was never their intention in the first place. They might just want to see the first week of a MOOC out of curiosity for the topic, but once they got an idea of the MOOC’s concept and the extent of the materials offered, their curiosity is satisfied and they drop out early on, as seen with the Dr Internet MOOC. Sadly, there is only so much that MOOC providers can do about this.

In conclusion, it has to be stated that even though the topic as well as the course design and requirements of the Dr Internet MOOC were designed to be low threshold, only a relatively small number of participants enrolled for the course. The narrative design patterns used to develop the MOOC did not have the intended effects on retention that the course creators had hoped they would have. Further research regarding the motivations for dropping out or finishing a course is an urgent desideratum – aspects like participant’s interest, required effort and perceived usefulness are not only important cognitive factors shaping the learning process, they also warrant a more in-depth consideration in the process of developing and implementing suitable design patterns to enhance motivation in the best possible way. However, the lack of sizeable results with regard to the design-based improvement of retention can be seen to raise other questions concerning the current prominent focus on dropout and completion rates in MOOCs. After all, these parameters largely stem from a more traditional educational context and might not always provide a fruitful perspective on inherently different learning platforms that result in diverging learning behaviors. A different, less problematizing focus both in research and practice could draw on the strengths of online learning environments rather than their weaknesses, and retention rates might benefit from this approach nonetheless.

Figures

Instructional triangle

Figure 1

Instructional triangle

Participant numbers (based on any activity and quizzes) from the Dr Internet MOOC

Figure 2

Participant numbers (based on any activity and quizzes) from the Dr Internet MOOC

Video schedule for the Dr Internet MOOC

Note

1.

The original questionnaire is available upon request, please contact the primary author.

References

Agarwal, A. (2016), “The global MOOC moment: massive open online courses, though controversial, are improving lives all over the world”, US News, February 1, available at: www.usnews.com/news/best-countries/articles/2016-02-01/the-global-mooc-moment

Bakker, E.J. (1993), “Activation and preservation: the interdependence of text and performance in an oral tradition”, Oral Tradition, Vol. 8 No. 1, pp. 5-20.

Bauer, R. and Baumgartner, P. (2012), “Showcase of learning: towards a pattern language for working with electronic portfolios in higher education”, Proceedings of the 16th European Conference on Pattern Languages of Programs, pp. E1-E30.

Benigeri, M. and Pluye, P. (2003), “Shortcomings of health information on the internet”, Health Promotion International, Vol. 18 No. 4, pp. 381-368.

Bonk, C.J., Lee, M.M., Reeves, T.C. and Reynolds, T.H. (in press), “The emergence and design of massive open online courses (MOOCs)”, in Reiser, R.A. and Demsey, J.V. (Eds), Trends and Issues in Instructional Design and Technology, 4th ed., Pearson, Boston, MA.

Bonk, C.J., Lee, M.M., Kou, X., Xu, S. and Sheu, F.-R. (2015), “Understanding the self-directed online learning preferences, goals, achievements, and challenges of MIT openCourseWare subscribers”, Educational Technology & Society, Vol. 18 No. 2, pp. 349-368.

Branson, R.K., Rayner, G.T., Cox, J.L., Furman, J.P., King, F.J. and Hannum, W.H. (1975), Interservice Procedures for Instructional Systems Development: Executive Summary and Model, Vols 1-5, TRADOC Pam 350-30, Army Training and Doctrine Command, Ft Monroe, VA.

Brodie, M., Flournoy, R.E., Altman, D.E., Blendon, R.J., Benson, J.M. and Rosenbaum, M.D. (2000), “Health information, the internet, and the digital divide”, Health Affairs, Vol. 19 No. 6, pp. 255-265.

Colman, D. (2013), “MOOC interrupted: top 10 reasons our readers didn’t finish a massive open online course”, Open Culture, available at: www.openculture.com/2013/04/10_reasons_you_didnt_complete_a_mooc.html (accessed August 24, 2016).

Dillahunt, T., Wang, Z. and Teasley, S.D. (2014), “Democratizing higher education: exploring MOOC use among those who cannot afford a formal education”, International Review of Research in Open and Distributed Learning, Vol. 15 No. 5, pp. 177-196.

Ebner, M., Scerbakov, A. and Kopp, M. (2015), “All about MOOCs”, in Jost, P. and Künz, A. (Eds), Digitale Medien in Arbeits- und Lernumgebungen, Pabst, Lengerich, pp. 148-155.

Ebner, M., Lorenz, A., Lackner, E., Kopp, M., Kumar, S., Schön, S. and Wittke, A. (2016), “How OER enhance MOOCs – a perspective from German-speaking Europe”, in Jemni Kinshuk, M. and Khribi, M.K. (Eds), Open Education: From OERs to MOOCs, Springer, Berlin, pp. 205-220.

Fidalgo-Blanco, A., Sein-Echaluce, M.L., García-Peñalvo, F.J. and Escaño, J.E. (2014), “Improving the MOOC learning outcomes throughout informal learning activities”, Proceedings of the Second International Conference on Technological Ecosystems for Enhancing Multiculturality, ACM, New York, NY, pp. 611-617.

Flath, H. (2012), “Storytelling im Journalismus. Formen und Wirkungen narrativer Berichterstattung”, doctoral’s thesis, TU Ilmenau, Ilmenau, available at: www.db-thueringen.de/servlets/DerivateServlet/Derivate-27890/ilm1-2013000242.pdf

Genette, G. (1972), Figures III, Ed. du Seuil, Paris.

Goodyear, P. (2005), “Educational design and networked learning: patterns, pattern languages and design practice”, Australasian Journal of Educational Technology, Vol. 21 No. 1, pp. 82-101.

Guàrdia, L., Maina, M. and Sangrà, A. (2013), “MOOC design principles: a pedagogical approach from the learner’s perspective”, eLearning Papers, Vol. 33 No. 4, pp. 1-6, available at: http://r-libre.teluq.ca/596/1/In-depth_33_4.pdf

Hansen, J.D. and Reich, J. (2015), “Democratizing education? Examining access and usage patterns in massive open online courses”, Science, Vol. 350 No. 6252, pp. 1245-1248.

Hollands, F.M. and Tirthali, D. (2014), “MOOCs: expectations and reality”, Full report, Center for Benefit-Cost Studies of Education, Teachers College, Columbia University, New York, NY, available at: http://cbcse.org/wordpress/wp-content/uploads/2014/05/MOOCs_Expectations_and_Reality.pdf

Jansen, D. and Schuwer, R. (2015), “Institutional MOOC strategies in Europe”, Status report based on a Mapping Survey Conducted in October-December 2014, EADTU/Lifelong Learning Programme, Heerlen, available at: www.surfspace.nl/media/bijlagen/artikel-1763-22974efd1d43f52aa98e0ba04f14c9f3.pdf

Jasnani, P. (2013), “Designing MOOCs”, A white paper on Instructional Design for MOOCs, available at: www.tatainteractive.com/pdf/Designing_MOOCs-A_White_Paper_on_ID_for_MOOCs.pdf (accessed August 24, 2016).

Khalil, H. and Ebner, M. (2013a), “How satisfied are you with your MOOC?” – a research study on interaction in huge online courses”, Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, AACE, Chesapeake, VA, pp. 830-839.

Khalil, H. and Ebner, M. (2013b), “Interaction possibilities in MOOCs – how do they actually happen?”, International Conference on Higher Education Development, Mansoura University, Mansoura, pp. 1-24, available at: http://de.scribd.com/doc/134249470/Interaction-Possibilities-in-MOOCs-%E2%80%93-How-Do-They-Actually-Happen (accessed August 24, 2016).

Khalil, H. and Ebner, M. (2014), “MOOCs completion rates and possible methods to improve retention – a literature review”, Proceedings of World Conference on Educational Multimedia, Hypermedia and Telecommunications, AACE, Chesapeake, VA, pp. 1236-1244.

Koening, J.M. and Zorn, C.R. (2002), “Using storytelling as an approach to teaching and learning with diverse students”, Journal of Nursing Education, Vol. 41 No. 9, pp. 393-399.

Koller, D., Ng, A., Do, C. and Chen, Z. (2013), “Retention and intention in massive open online courses: in depth”, available at: http://er.educause.edu/articles/2013/6/retention-and-intention-in-massive-open-online-courses-in-depth (accessed August 24, 2016).

Kopp, M. and Lackner, E. (2014), “Do MOOCs need a special instructional design?”, EDULEARN14 Proceedings, pp. 7138-7147.

Lackner, E., Ebner, M. and Khalil, M. (2015), “MOOCs as granular systems: design patterns to foster participant activity”, eLearning Papers, Vol. 42 No. 3, pp. 28-37, available at: www.openeducationeuropa.eu/sites/default/files/legacy_files/asset/Design_Patterns_for_Open_Online_Teaching_and_Learning_In-Depth_42_3.pdf

Lackner, E., Kopp, M. and Ebner, M. (2014), “How to MOOC? – a pedagogical guideline for practitioners”, in Roceanu, I. (Ed.), Proceedings of the 10th International Scientific Conference “eLearning and Software for Education”, Editura Universitatii Nationale de Aparare “Carol I”, Bucharest, pp. 215-222.

LiteraryDevices Editors (2016), Cliffhanger, available at: http://literarydevices.net/cliffhanger/ (accessed August 24, 2016).

Neuböck, K., Kopp, M. and Ebner, M. (2015), “What do we know about typical MOOC participants? First insights from the field”, Proceedings of eMOOCs 2015 Conference, pp. 183-190.

Nipper, K. and Sztajn, P. (2008), “Expanding the instructional triangle: conceptualizing mathematics teacher development”, Journal of Mathematics Teacher Education, Vol. 11 No. 4, pp. 333-341.

Pappano, L. (2012), “The year of the MOOC”, The New York Times, November 2, available at: www.nytimes.com/2012/11/04/education/edlife/massive-open-online-courses-are-multiplying-at-a-rapid-pace.html?_r=0 (accessed November 2, 2012).

Robertson, N., Polonsky, M. and McQuilken, L. (2014), “Are my symptoms serious Dr Google? A resource-based typology of value co-destruction in online self-diagnosis”, Australasian Marketing Journal, Vol. 22 No. 3, pp. 246-256.

Sadik, A. (2008), “Digital storytelling: a meaningful technology-integrated approach for engaged student learning”, Educational Technology Research and Development, Vol. 56 No. 4, pp. 487-506.

Salpeter, J. (2005), “Telling tales with technology: digital storytelling is a new twist on the ancient art of the oral narrative”, Technology & Learning, Vol. 25 No. 7, pp. 18-24.

Santos, J.L., Charleer, S., Parra, G., Klerkx, J., Duval, E. and Verbert, K. (2013), “Evaluating the use of open badges in an open learning environment”, in Hernández-Leo, D., Ley, T., Klamma, R. and Harrer, A. (Eds), Scaling up Learning for Sustained Impact. Proceedings of the 8th European Conference, on Technology Enhanced Learning, Springer-Verlag, Berlin Heidelberg, pp. 314-327.

Shah, D. (2015), “MOOC trends in 2015: rise in self-paced courses: how have MOOC platforms and MOOC takers alike reacted to self-paced?”, Class Central, December 9, available at: www.class-central.com/report/mooc-trends-2015-rise-self-paced-courses/ (accessed August 24, 2016).

Siemens, G. (2004), “Connectivism: a learning theory for the digital age”, available at: www.elearnspace.org/Articles/connectivism.htm (accessed August 24, 2016).

Spath, C. and Foerg, B. (2005), Storytelling & Marketing, Echomedia, Wien.

Song, D. and Bonk, C.J. (2016), “Motivational factors in self-directed informal learning from online learning resources”, Cogent Education, Vol. 3 No. 1, p. 1205838.

University of Houston (2016), “The 7 elements of digital storytelling”, available at: http://digitalstorytelling.coe.uh.edu/page.cfm?id=27&cid=27&sublinkid=31 (accessed August 24, 2016).

Veletsianos, G., Collier, A. and Schneider, E. (2015), “Digging deeper into learners’ experiences in MOOCs: participation in social networks outside of MOOCs, notetaking, and contexts surrounding content consumption”, British Journal of Educational Technology, Vol. 46 No. 3, pp. 570-587.

Veletsianos, G., Reich, J. and Pasquini, L.A. (2016), “The life between big data log events learners’ strategies to overcome challenges in MOOCs”, AERA Open, Vol. 2 No. 3, pp. 1-10, available at: http://ero.sagepub.com/content/2/3/2332858416657002 (accessed November 25, 2016).

Wüster, M. and Ebner, M. (2016), “How to integrate and automatically issue open badges in MOOC platforms”, Proceedings of the European Stakeholder Summit on Experiences and Best Practices in and Around MOOCs, pp. 279-286.

Zimmermann, C., Kopp, M. and Ebner, M. (2016), “How MOOCs can be used as an instrument of scientific research”, Proceedings of the European Stakeholder Summit on Experiences and Best Practices in and Around MOOCs, pp. 393-400.

Zimmermann, C., Lackner, E. and Dreisiebner, G. (2017), “Designing a MOOC to foster critical thinking and its application in business education”, International Journal of Business Education, (under review).

Acknowledgements

The authors would like to thank Mohammad Khalil for preparing the data collected via learning analytics for the case study MOOC.

Corresponding author

Martin Ebner can be contacted at: martin.ebner@tugraz.at

Related articles