Observations from three years of online pandemic learning response on OpenWHO

Abstract

Purpose

OpenWHO is the World Health Organization's online learning platform that was launched in 2017. The COVID-19 pandemic led to massive growth in the number of courses, enrolments and reach of the platform. The platform is built on a stable and scalable basis that can host a large volume of learners. The authors aim to identify key factors that led to this growth.

Design/methodology/approach

In this research paper, the authors examined OpenWHO metadata, end-of-course surveys and internal processes using a quantitative approach.

Findings

OpenWHO metadata showed that the platform has hosted over 190 health courses in 65 languages and over seven million course enrolments. Since the onset of the pandemic, there have been more women, older people and people from middle income countries accessing courses than before. Following data analysis of the platform metadata and course production process, it was found that several key factors contributed to the growth of the platform. First, OpenWHO has a standardised course production pathway that ensures efficiency, consistency and quality. Further, providing courses in different languages increased its reach to a variety of populations throughout the world. For this, multi-language translation is achieved through a network of translators and an automated system to ensure the efficient translation of learning products. Lastly, it was found that access was promoted for learners with disabilities by optimising accessibility in course production. Data analysis of learner feedback surveys for selected courses showed that the courses were well received in that learners found it useful to complete courses that were self-paced and flexible. In addition, results indicated that preferred learning methods included videos, downloadable documents, slides, quizzes and learning exercises.

Originality/value

Lessons learnt from the WHO's learning response will help prepare researchers for the next health emergency to ensure timely, equitable access to quality health knowledge for everyone. Findings of this study will provide valuable insights for educators, policymakers and researchers in the field who intend to use online learning to optimise knowledge acquisition and performance.

Citation

Utunen, H., Appuhamy, R., Attias, M., Ndiaye, N., George, R., Arabi, E. and Tokar, A. (2023), "Observations from three years of online pandemic learning response on OpenWHO", International Journal of Information and Learning Technology, Vol. 40 No. 5, pp. 527-540. https://doi.org/10.1108/IJILT-12-2022-0220

Introduction

The COVID-19 pandemic has changed the way we teach and learn. Public health measures that were necessary to control the pandemic, such as travel restrictions, physical distancing and lockdowns disrupted formal and informal face-to-face learning interactions globally. This resulted in a shift to distance education using remote digital learning solutions.

In June 2017, the World Health Organization (WHO) launched its online learning platform, OpenWHO.org to transfer scientific, technical and operational knowledge to frontline responders and the public. It provides free, self-paced, multilingual courses accessible in low-bandwidth and offline formats. After serving frontline responders in regionalised outbreaks from Ebola to plague, the platform scaled up course production for the COVID-19 pandemic. As of January 2023, there are 190 courses on various health topics in 65 languages with 7.4 million enrolments.

The OpenWHO production team supports the technical content creators, who are subject-matter experts, during all stages of the production cycle. The team ensures that the courses embody adult learning principles, are engaging, cover the learning objectives and drive learners towards course completion. Quality is maintained by guaranteeing that content is derived from the most up-to-date scientific, technical and operational knowledge and that it is aligned with relevant WHO technical documents and guidelines.

We aimed to identify key factors that affected the reach, scalability and choice of learning methods on an online health learning platform designed to reach a mass audience.

To do that, we examined the course production processes, usage data and learner surveys to identify factors that contributed to the platform's massive growth over the past three years. We also explored the impact of design and development strategies and the WHO's learning response to rapidly, efficiently and equitably share health knowledge via the platform so that we are better prepared to respond to a possible next pandemic via the online dissemination of learning from WHO experts to the world.

By analysing the experiences and outcomes of learners on a large-scale online education platform with over seven million enrolled learners, this paper aims to contribute to a better understanding of how online education can be effectively delivered at scale, and provide valuable insights for educators, policymakers, and researchers in the field.

Online learning and MOOCs

There has been extensive research on massive open online courses (MOOCs) – an asynchronous format of online learning for a mass audience – indicating its role in democratising education (Jansen et al., 2015; Kurt, 2018; Lugton, 2012; Shah and Pickard, 2019). Despite the advantages of MOOCs, such as flexible and independent learning which allows learners to learn any time, anywhere (Carapeto and Vieira Barros, 2019), studies have found a decline in course completion in most MOOCs (Dridi et al., 2020; Howland and Moore, 2002; Petrides, 2002; Vonderwell, 2003). This has been attributed to the design and format of MOOCs which do not consider low-resource locations, learners with disability, or digital literacy (Dridi et al., 2020; Moser-Mercer, 2014).

Further, most research on MOOCs has been conducted in university settings not indicating if these interventions would be beneficial for all types of learners regardless of their age, profession, gender and geographical locations (Schulze et al., 2017). As most MOOCs have been commonly delivered in English, it may have hindered inclusive education (Skutnabb-Kangas, 2000). To date, there is minimal available data that has examined MOOCs considering all those barriers to make learning more accessible.

Online learning for accessibility and adaptability

Access to accurate health information and sharing the acquired knowledge are pivotal in improving the health of communities. While technology advancement and online learning may facilitate access to evidence-based learning (Mehta et al., 2013), there may be many underprivileged communities who cannot access health information due to access or usage issues (Attewell, 2001). This requires the teams who design, develop and deliver learning interventions to give careful consideration to learner demographics, their socioeconomic status and their disabilities. Previous research reports have indicated that more affluent communities can access or use computers (Boser, 2013).

Whereas learning technologies have closed resource gaps between learners in different environments (Hansen and Reich, 2015), there are inherent limitations in Internet connection and digital literacy. Reports have shown that refugee camps or low-resourced settings have limited access to broadband Internet connection (Dridi et al., 2020; Moser-Mercer, 2014), thus, mobile phones are commonly used for learning (Wildavsky, 2017). Therefore, creating low-bandwidth learning material can facilitate access to or adapt and share the learning material.

Creating inclusive education and digital equity requires looking beyond technology limitations to consider language or physical barriers as well (Skutnabb-Kangas, 2000; Nwokediuko, 2012). Evidence has shown that providing learning material in the native language of a community can increase learners' comprehension of the information, thus, affecting the knowledge acquisition and information sharing in health emergencies (Translators Without Borders, 2015). In addition, disability is an essential component of addressing inequities perpetuated by inaccessible technology-enabled environments (Fennelly-Atkinson et al., 2022). Despite the proven advantages of online learning, the accessibility of online learning materials impedes disabled learners from benefiting fully and equally (Laufer Nir and Rimmerman, 2018). In that regard, inaccessibility preserves a digital divide that excludes disabled people as technology advances (Ferri and Favalli, 2018). Creating accessible online learning environments is central to any design and development efforts to accommodate the environment so everyone can participate fully.

Research methodology

We used a multi-source quantitative approach for this study. The study was designed to address the following research question.

RQ.

What are the key factors that affect the scalability of online courses to reach a mass audience and underrepresented groups such as low or middle-income countries, older people and women?

Several data sources guided us to address the research question. These included (1) OpenWHO metadata, (2) internal reviews of existing practices and processes and (3) end-of-course surveys.

Anonymised metadata on OpenWHO platform use, including enrolment trends, certificate attainment, completion rates and the number of courses and languages were obtained from the platform's in-built data sets and reporting system. Descriptive analysis using the Microsoft® Power BI tool was used to analyse the metadata. Key outcome variables of interest were users' self-reported data such as location, gender, language, age and affiliation, along with course-based statistics such as the number of courses enrolled and completion rates. We analysed user patterns and locations based on Google Analytics.

Reviews of internal processes for course production and publishing were done by the authors with expertise in public health, learning and health emergencies. This involved reviewing, evaluating and interpreting data collected from multiple sources including formal and informal feedback from content creators and learners.

We utilised data from end-of-course surveys which were administered to all learners who had completed selected courses. These surveys were validated by an expert panel for its face validity and piloted before officially being administered to all learners. Descriptive analysis was used to analyse data from the surveys.

Findings

We present the results of the analysis of OpenWHO metadata, internal reviews and survey in the following sections.

OpenWHO metadata

OpenWHO was launched in June 2017. It initially hosted courses designed to manage disease outbreaks occurring at that time. These included Ebola (in the Democratic Republic of the Congo), Pneumonic plague (in Madagascar) and Diphtheria (in Cox Bazar, Bangladesh). In the first year, there were 49,000 enrolments, and by January 2020, the number had increased to 160,000.

In January 2020, OpenWHO launched the “Introduction to COVID-19” course, and within the first three months, the course had 232,890 enrolments and was translated into 13 languages (Utunen et al., 2020a). On average, 53% of those who enrolled in the introductory course were new to the OpenWHO platform. In the first 10 weeks following the pandemic declaration, learners used the learning platform most in countries with the highest COVID-19 cases (Utunen et al., 2021a) (see Figure 1).

Before the launch of the first COVID-19 course in January, traffic to OpenWHO stood at 300–1,000 requests per minute (rpm). By February 2020, that had more than doubled to 2000–2,500 rpm. At its highest that year, traffic to the platform exceeded 30,000 rpm, with relative lows of 15,000 rpm during idle phases. OpenWHO's server bandwidth required an equally significant increase, tenfold from 100 to 1,000 megabits per second (Mbit/s), to manage the influx of traffic. To supplement this, OpenWHO's platform provider the Hasso Plattner Institute (HPI) added multiple virtual servers to the OpenStack based-private cloud infrastructure that hosts the platform (George et al., 2021).

The influx of new learners resulted in an acceleration in traffic to OpenWHO, resulting in a corresponding increase in requests for technical support through the OpenWHO 24/7 help desk. To manage this, HPI and WHO developed a support chatbot powered by artificial intelligence (AI) (George et al., 2021). Early data showed that 40% of learners who interacted with the chatbot were satisfied with its response and did not need to submit their question as a ticket to the help desk (WHO, 2022).

Currently, there are 45 other COVID-19-related courses on the platform. In addition, from 2020 to 2022, OpenWHO added more than 140 courses on various health topics (including courses on 32 other infectious diseases) to the learning platform. The total number of enrolments on OpenWHO increased to 4.8 million by January 2021 and to more than 7 million by January 2023.

All countries around the world have accessed the OpenWHO courses. Learners from low- and middle-income countries increased from half to nearly three-quarters in 2021. This increase was driven by the rise in overall enrolments from middle-income countries, which went from 40.2% to 70.6%. This was accompanied by a drop in lower-income countries from 14% to 3.4% and in higher-income countries from 45.6% to 26% (Utunen et al., 2021b).

There was an increase in the proportion of women enrolling in courses from 40% before the COVID-19 pandemic to 50%. There was also a shift in the age of learners on the platform. The proportion of learners who were 70 years of age and over grew from zero before the pandemic to 4.6%. The percentage of learners less than 20 years of age increased from 1.26% before the pandemic to 9.7%. Overall completion rates for courses on OpenWHO increased from 39% before the pandemic to 54% during the pandemic.

An analysis of quiz scores of COVID-19 vaccine courses from mid-December 2020 to mid-April 2021 showed that overall learners averaged 76% on a pre-course quiz compared to 85% post-course quiz (Goldin et al., 2021).

Internal reviews

In this section, we present the results of the internal reviews for course production and translation processes, as well as the platform features.

Course production process

OpenWHO adopted a course production cycle that is fit for WHO emergencies learning content development (see Figure 2). The transformation of technical knowledge into an online course involves six steps (Gamhewage et al., 2020). These include identifying a training need, creating the course, refining the content, clearing the course with WHO experts, publishing on the platform and updating and improving the learning material (Gamhewage et al., 2020).

With respect to training needs analysis, the required knowledge and skills to respond to and manage health emergencies were identified through WHO subject-matter experts, requests by member states, surveillance reports, or experience in past epidemics or pandemics. The training needs analysis was a crucial step in the course creation cycle to shape the entire process yielding relevant courses. Further, a review of course materials showed that multiple means of representation of the E-learning content was used in the online courses to cater to different learner needs and contexts. This included presentation slides, presentations with audio voiceover, instructional videos and quizzes.

Videos could be streamed or downloaded in high- or low-definition with subtitles. Audio files, transcripts and presentation slides could also be downloaded and viewed offline. However, there were some restrictions in a number of countries for the video hosting service used by OpenWHO. For this reason, the video hosting platform was changed to one that broadcasted videos through an international Content Delivery Network (CDN) to be able to reliably deliver content to viewers worldwide.

Providing up-to-date information is critical in a health emergency, especially when the evidence can change as information becomes available. Therefore, published courses were updated if the scientific evidence, guidelines, or policies changed significantly. For instance, the “Introduction to COVID-19” course has been revised 13 times since its launch (Utunen et al., 2021c).

Multi-language translation

The platform currently hosts courses in 65 languages, including the official languages of all WHO regions and 44 of the 46 official languages of the least-developed countries. A total of 20 million words have been translated; each course, on average, has been translated into 4.8 languages. The most used language has been English, followed by Spanish, French, Arabic, Portuguese, Indian sign language, Hindi, Indonesian, Russian and Italian.

The internal review of the translation process indicated that the OpenWHO team works with several translation solutions in a network of volunteer translators from WHO country and regional offices and other entities to translate course presentations and other core learning content. Automated translation is also used, which has allowed the near-simultaneous availability of video subtitles in an array of languages. The OpenWHO team relies on a custom-built automatic transcription and translation tool embedded into the OpenWHO course publishing interface. The application, Transpipe, is the product of a long-term collaboration between WHO and OpenWHO's platform provider HPI with pro-bono support from Amazon Web Services, which provided the product based on which Transpipe was built. Powered by machine learning, Transpipe continuously augments its vocabulary bank to improve transcription and translation accuracy.

Promoting accessibility

A review of courses on the platform in terms of accessibility highlighted the launch of the “Introduction to COVID-19” course in Indian sign language in March 2020. At the time of this study, there were more than 55,000 learners enrolled in this course. It was not only in the top three courses taken up by learners in India, but was among the top language versions used in the United States of America and China (Utunen et al., 2020b).

Further, in early 2022, a typeface called “Atkinson Hyperlegible” was progressively rolled out for material produced by OpenWHO. This typeface was developed for the Braille Institute and focuses on letterform distinction to improve legibility for people with low vision (Braille institute, 2022). In addition, OpenWHO has several features to enhance accessibility for visually impaired learners. This includes text alternatives for non-text content, high-contrast colour schemes and downloadable text items (e.g. slides, transcripts and supporting materials). Downloaded materials can be explored in the appropriate resolution or size on a different device or be printed.

User interface

The OpenWHO interface has several features that enable users to locate and access relevant learning material, including categorising courses by topic and language and into thematic channels to make courses easier to find. The “Serving Countries” portal has resources organised by country. Learners can find country-specific information within those channels in their national languages that have been developed and curated in collaboration with the relevant country offices and ministries of health.

End-of-course surveys

In this section, we present the survey results of key OpenWHO courses. and elaborate how they informed the outcomes of the pandemic learning.

Integrated disease surveillance and response course (IDSR)

For the IDSR English course, 4,345 learners were invited to take part in a feedback survey; 375 responses were received, with a response rate of 8.6%. The top five preferred learning methods identified by respondents were videos, downloadable documents, slides, quizzes and learning exercises (see Figure 3).

COVID-19 vaccination courses

OpenWHO administered a learner feedback survey to evaluate a vaccine training course. From December 2020 to April 2021, 53,595 learners were enrolled in the course, and 2019 participants responded to the survey (Goldin et al., 2021). The survey responses showed that 65.6% (n = 1,293) of respondents preferred online training. The reasons that they indicated included completing a course any time at their convenience (n = 1,204, 59.6%), the self-paced nature (n = 1,039, 51.5%), the ability to download the materials (n = 907, 44.9%), the ability to replay sections (n = 890, 44.1%) and the increased ability to concentrate (n = 520, 25.8%).

In 2022, WHO and its partners distributed a survey in English and French to identify which online learning formats were preferred by immunisation staff across the globe, and 1,132 people responded (French 553, English 579). On the question “Which online learning course formats do you find most useful?”, the most popular answer was “Courses that I can take at my own time and pace” (see Figure 4).

Infection prevention and control and introduction to COVID-19 courses

OpenWHO administered a survey to enrollees in the English and Spanish versions of Infection Prevention and Control (IPC) and Introduction to COVID-19 courses on 7 August 2020. The survey was closed on 31 August 2020, and 23,279 responses were received. Four questions were designed to collect feedback on the learners' learning experience during the COVID-19 pandemic.

These two courses accounted for more than one-third of all enrolments across COVID-19 courses. A total of 98.2% said the course met their learning needs. A total of 98.6% indicated the language used in the learning materials was easy to understand, and 98.1% said they could navigate the learning content on the website (see Figure 5).

Data triangulation from the platform metadata, internal reviews and end-of-course surveys, indicated five key factors that led to the massive growth and equitable reach of the platform. These include.

1. Anticipating learning needs: As WHO's online learning platform offers courses on health emergencies, it is crucial that the courses address the knowledge and skills needed to respond to and manage health emergencies effectively. This was reflected by the demand for these courses being highest among health care workers and in countries with the highest number of COVID-19 cases. The COVID-19 courses also fuelled a rise in enrolments across other courses in the platform.

2. Timeliness: Courses on health emergencies must address learning needs promptly. Real-time, just-in-time learning takes the approach of providing instantaneous evidence-based learning content to respond to an emergency. For instance, OpenWHO published the introductory COVID-19 course on 26 January 2020, four days before the declaration of COVID-19 as a public health emergency of international concern (PHEIC).

3. Quality: The review of processes showed that the teams are required to employ high-quality content and incorporate sound instructional design principles to meet the learning goals. All teams were to employ the standardised course production process to ensure that OpenWHO consistently maintained the quality across each course published on the platform. As the United Nations health agency, WHO is an authoritative voice for public health and has the responsibility to ensure the content reflects the best information available for learners at the time, presented in a clear and compelling way.

4. Adaptability: Online courses on OpenWHO that offer alternative ways to access the content have catered to different learner needs and contexts. Feedback from user surveys showed that learners found the self-paced and flexible nature of the courses useful. This also facilitated a learning multiplier effect in that the online learning resources were transformed, modified and adapted into other formats to suit the local context and learning needs.

5. Accessibility: An essential consideration was given to the target audience's unhindered access to the learning material. The pandemic expanded OpenWHO learning to previously underrepresented groups, including women and learners aged over 70 and younger than 20, advancing equity in access to knowledge. Providing the courses for free in multiple languages, with user-friendly platform features and designing for people with disabilities is likely to have increased the uptake of the courses.

Discussion

The primary goal of this study was to identify the factors that led to the growth of the OpenWHO platform in terms of its reach to a global audience. In this respect, this study aimed to democratise education by making learning available to all individuals regardless of their backgrounds and circumstances to the maximum extent possible. Several factors contributed to the growth of the platform within a short period of time. We found from the OpenWHO metadata that offering courses in multiple languages and free of charge increased enrolments in both middle-income and low-income countries. As cost is a known barrier to access education (Utunen et al., 2020a; b; 2021a, b), providing free courses online can make learning more accessible to all learners regardless of their circumstances. Furthermore, multilingualism was a key factor as language availability is a critical component of learning accessibility. Language can be a crucial obstacle in health literacy (Nwokediuko, 2012). Providing health information in a native language has been proven effective in enhancing knowledge and comprehension (Perera et al., 2012; TWB, 2015, 2019). Language usage data demonstrated that regardless of where learners reside, they prefer to learn in their native languages (Utunen et al., 2020b).

In addition, the review of the course production process indicated that an efficient and effective approach based on learning sciences and universal design for learning (UDL), a thorough needs analysis, as well as close collaboration with stakeholders yielded accessible and equitable learning experiences regardless of age, profession and location. Careful consideration regarding user interface (UI) and user experience (UX) made it easy for learners to locate and access the learning material easily. Adhering to accessibility principles was an imperative part of the design and development of the courses. It is estimated that more than 1.5 billion people live with hearing loss (WHO, 2022), and at least 2.2 billion people have near or distant vision impairment (WHO, 2021). Indian sign language was offered on OpenWHO for the first time at the beginning of the COVID-19 pandemic and continues to have a high uptake. Design features such as using a typeface with high legibility, text alternatives, high-contrast colour schemes and downloadable items have improved accessibility for visually impaired people.

Another factor that played a role in the growth of the platform was providing just-in-time learning that meets learners' knowledge or performance gaps. Learners were able to download and adapt the content according to their contextual and community needs. Countries worldwide have used these materials for training fieldworkers, local dissemination, social networks, microlearning, adaptation to university courses, precision group targeting and hybrid models, bringing the knowledge to more learners than would have otherwise been reached directly through the learning platform (Utunen et al., 2022).

Lastly, the mode of delivery (i.e. online self-paced) played an important part in providing flexible learning anytime, anywhere. This mode of delivery was working for the needs of learners delivering health emergency response in health emergencies such as the COVID-19 pandemic with a demonstrable gain in knowledge (Goldin et al., 2021).

Implications

Findings of this study demonstrated that making online learning scalable, accessible and equitable should take a holistic approach. This includes identifying learning needs, employing sound pedagogy and instructional strategies, leveraging technology for enhancing learning and adhering to accessibility principles to fulfil digital equity. This study provides evidence that employing a systematic approach to design, develop and deliver learning material that facilitates accessibility to adapt and share health information is central in providing online health emergency training.

While an increase in enrolments can indicate the degree to which the platform has optimally achieved its goal to deliver health information, further research could explore the effectiveness of learning design approach through more robust knowledge assessment. Additionally, further research could examine learner inclusion based on the social disability model, which focuses on social responsibility to identify barriers and bridge the digital divide commonplace in most MOOCs.

Limitations

The limitations include the constraints of the platform data collection system which is dependent on individuals' self-identifying their ages, affiliations and other demographics at the moment of their registration. Learners may choose not to report this information and may not return to update their data as it evolves during their learning journey. The end-of-course surveys were also limited to learners in specific courses selected and, as they are optional, to those who choose to respond, which may favour individuals who have the strongest opinions about the course or have more time available for participation. With more than 7 million enrolments, a platform-wide survey across all courses was not feasible. Another potential limitation of the study was that the key learnings of the study were based on the analysis and interpretation of the results of the authors. Finally, as the internal review was conducted by some members of the research team, biases in the retrospective review of processes and conclusions should not be excluded.

Conclusion

The COVID-19 pandemic challenged us to look beyond the traditional ways of knowledge transfer to adapt, upscale rapidly and utilise technology and innovations to ensure that health information reached learners in an efficient, effective and timely manner. To achieve this, WHO had to transform how it traditionally provided learning content online. It had to adapt to the changing landscape to equip staff, health workers and the public with the health knowledge required to fight the ongoing pandemic.

Prioritising access to learning has enabled OpenWHO to have tremendous and equitable reach globally. By anticipating learning needs, timeliness, quality, adaptability, scalability and accessibility, OpenWHO was able to reach millions of learners. However, there are still gaps in getting health information to where it is most needed. They include challenges in reaching people with limited access to technology, limited Internet access and those facing barriers to digital literacy.

We now have a set of robust tools and resources that we will be able to build upon and use beyond the pandemic. This would help prepare us for the next pandemic and to ensure that everyone has equal access to quality health knowledge.

Figures

Figure 1

Growth of enrolments, courses and critical achievements of OpenWHO since 2020

Figure 2

Production cycle on OpenWHO

Figure 3

Preferred learning methods selected by respondents to the integrated disease surveillance and response (IDSR) survey

Figure 4

Responses from immunisation staff to the question “which online learning course formats do you find most useful?”

Figure 5

IPC and COVID-19 course learners in 2020 responding to the question of whether the course met the learning needs