The research trends in recommender systems for e-learning: A systematic review of SSCI journal articles from 2014 to 2018

Abstract

Purpose

A recommendation algorithm is typically applied to speculate on users’ preferences based on their behavioral characteristics. The purpose of this paper is to provide a systematic review of recommendation systems by collecting related journal articles from the last five years (i.e. from 2014 to 2018). This paper aims to study the correlations between recommendation technologies and e-learning systems.

Design/methodology/approach

The paper reviews the relevant articles using five assessment aspects. A coding scheme was put forward that includes the following: the metrics for the e-learning system, the evaluation metrics for the recommendation algorithms, the recommendation filtering technology, the phases of the recommendation process and the learning outcomes of the system.

Findings

The research indicates that most e-learning systems will adopt the adaptive mechanism as a primary metric, and accuracy is a vital evaluation indicator for recommendation algorithms. In existing e-learning recommender systems, the most common recommendation filtering technology is hybrid filtering. The information collection phase is an important process recognized by most studies. Finally, the learning outcomes of the recommender system can be achieved through two key indicators: affections and correlations.

Originality/value

The recommendation technology works effectively in closing the gap between the information producer and the information consumer. This technology could help learners find the information they are interested in as well as send them a valuable message. The opportunities and challenges of the current study are discussed; the results of this study could provide a guideline for future research.

Citation

Zhong, J., Xie, H. and Wang, F.L. (2019), "The research trends in recommender systems for e-learning: A systematic review of SSCI journal articles from 2014 to 2018", Asian Association of Open Universities Journal, Vol. 14 No. 1, pp. 12-27. https://doi.org/10.1108/AAOUJ-03-2019-0015

Introduction

E-learning is defined as an instruction tool that provides knowledge and helps facilitate learning by use of a digital device or a web technology (Clark and Mayer, 2016). In comparison with traditional instruction modes and learning approaches, the use of e-learning is more effective for learning purposes. For example, early research (McClusky, 1947) showed that the educational film, as a delivery medium on an e-learning system, has better learning outcomes because it contributes to achieving educational goals. Further, e-learning is a type of new learning style that could be a solution for lifelong personal learning (Zhang et al., 2004), as learners can learn without the limitations of time and place. Fundamentally, it is of great value for educational reform and development. To be specific, e-learning is widely regarded as “educational technology, information and communication technology (ICT), multimedia learning, technology-enhanced learning (TEL), computer-based instruction (CBI), a virtual learning environment (VLE), mobile learning” (Ridwan, 2015) and so on. With the progress of information and communication technology development, e-learning system provides support for educational management. Due to the importance of e-learning system, many researchers have expended much effort on the research of emerging educational technologies on e-learning platforms through some theoretical models or frameworks.

Specifically, Persico et al. (2014) proposed an improved technology acceptance model (TAM) that is a three-dimensional model to evaluate the technological innovations of an online learning system, which includes three dimensions of “phases of use, users and components.” Arkorful and Abaidoo (2015) established the application of information and communication technology in teaching and learning processes, and the advantages and disadvantages in using e-learning in higher educational institutes through literature reviews of previous research. Truong (2016) put forward his views on ways to integrate learning styles into adaptive e-learning systems. His review paper describes predictions of online learning styles and a method of automatic classification is also proposed, as well as recommendations and potential opportunities for future work. Based on the TAM, a framework was proposed to investigate the effect of learning styles in blended e-learning systems (Al-Azawei et al., 2017). Although many variables could be integrated, the author chose the “integration of perceived satisfaction and technological acceptance,” and his findings suggested that learners’ needs, rather than their learning styles, should be used as a research variable for personalized educational systems. Big data technology has also attracted the interest of scholars and is a promising research topic for online learning systems. Huda et al. (2018) argued that the emerging technology of big data plays an important role in the innovative environments of online learning resources, which promotes a conducive learning environment and performance improvements during a student’s learning development process. The above studies on e-learning technologies introduce the trends and developments of the current educational technologies, especially the wide application of information and communication technologies in teaching and learning. The users’ learning style has become a hot topic in the study of learning system models in recent years. Based on a model of online learning style, Li et al. (2019) proposed a course recommendation method for learners. The conclusion indicates that learning style is an essential factor in the recommendation system as the course information can be recommended to the target users according to the data of their learning preferences.

From the perspective of information acquisition, searches and recommendations are the two primary means for users to get information (Alonso et al., 2006). Search is an active and explicit behavior, and the search engine is the best tool to satisfy the needs of users in finding matching information. However, dissemination and acquisition of information could also be achieved through a recommender system. In the recommendation process, the user passively receives what the system recommends. Often, the user’s needs are vague and unclear. Therefore, a recommender system tends to have significant individual differences due to different user requirements.

Moreover, the recommendation algorithm has a close relationship with the recommendation content. The process for users to obtain recommendations may be continuous and long-term, because the system needs to use different recommendation filtering techniques in order to provide users with appropriate content, such as product information or news, on the basis of understanding users’ interests and their behavior. Therefore, the recommendation technology could be applied in various fields, such as shopping websites, music channels, learning platforms and so on. The great potential for this technology appeals to many researchers who are increasingly focused on the application and the development of the recommendation technology in an e-learning system. In this paper, we review related recommendation technologies for e-learning systems. The journal articles that fall within the scope of both “recommendation” and “e-learning” are included in our research.

In recent years, the popularity of internet technology has made it easier for people to access information and facilitate their studies and their life. It has become a common phenomenon for many people to search for information and learn new skills through the internet. Over the past decade, the number of online learners has increased dramatically (Kim and Bonk, 2006). To cater to people’s needs, many online learning platforms have arisen. However, due to information overload issues (Hiltz and Turoff, 1985), it is not easy for users to find their favorite subject amidst all the information available. Second, the majority of people do not understand their own specific needs. The user efficiency for information retrieval will be reduced, so e-learning platforms need to adopt specific strategies to optimize decentralized resources. The application of a recommender technology is a good solution. The recommender system provides users with customized recommendations within a specific domain (Martínez et al., 2009) and presents useful information to interested users. Therefore, our research on online learning platforms is of great importance. It could not only contribute to optimizing recommendation strategies but also act as a guideline for future researchers to navigate the best research directions.

As shown in Figure 1, there was a total of 115 articles relevant to e-learning recommender systems from the years 2014 to 2018. For an in-depth analysis, 31 articles were selected from the following nine top-tier publications including Computers in Human Behavior, Computers Education, IEEE Transactions on Learning Technologies, Educational Technology Society, Interactive Learning Environments, British Journal of Educational Technology, Educational Sciences Theory Practice, Journal of Computer Assisted Learning and Journal of Computing in Higher Education. On average, 6.2 articles are published in these journals every year.

According to the citation analysis report of the Web of Science database¹, 31 articles have been cited 258 times, of which 255 self-citations have been removed. Figure 2 shows the citation frequency of these articles every year, which shows that the recommendation of e-learning has gradually become a hot research topic since the year 2014 and has attracted the attention of many researchers.

Serious Game is an important pedagogic tool that could be applied to the curriculum to promote students’ learning. Gauthier and Jenkinson (2018) applied the Activity Theory Model of Serious Game to analyze the relationship between game design decisions and teaching strategies. By comparing students’ learning behavior in a game and a nongame application, this article introduced three game design strategies that could enhance students’ productive negativity in STEM higher education. Through a qualitative and quantitative analysis, Wang et al. (2017) found that the mechanics and dynamics of Serious Game will affect the students’ concentration and enjoyment, while another journal demonstrated four kinds of relationships between users’ learning styles and the game genre, two of which have proven to be valid and can be utilized in e-learning recommender systems based on the special needs and the preferences of learners (Khenissi et al., 2016).

The Semantic Web is an intelligent network which can be understood by people and machines, as well as providing automatic services depending on the semantics in the e-learning system (Sampson et al., 2004). With the development of semantic web technology, Ouf et al. (2017) discussed a new personalized learning framework for an e-learning ecosystem, and a complete learning package consisting of learner models and learning process components was created as well, while Albatayneh et al. (2018) introduced a recommendation architecture for an e-learning forum on the basis of semantic filtering technology and learners’ negative ratings. Moreover, this learning recommender system performed well and had a better recommendation accuracy than similar systems. A recommendation approach was presented based on a user similarity calculation. The method used various technologies, such as the semantic web technology, to customize the best learning path for learners on the basis of students’ preferences and interests. It has also proven to be effective in the social learning environment (Halimi et al., 2014).

Over the past years, the issue of the social network has appealed to many scientists’ interests, as network theory can connect related or nonrelated metadata and can be used to analyze social phenomena (Borgatti et al., 2009). In addition, the social network is crucial to collaboration and communication between people. People also use tags to indicate their characteristics, hobbies and so on, and these tags are also used to share photos or texts. Collaboration tags are very popular on the internet and could be used to describe the process by which users, using keywords, add metadata to share content (Golder and Huberman, 2005). Klašnja-Milićević et al. (2018) use the user’s tags to improve the recommendation algorithm in the e-learning environment and integrate social collaborative tagging technology to provide recommendations of learning materials for users on the online learning platform. Karataev and Zadorozhny (2017) exploited a self-adaptive learning framework for the teaching process and applied it to collective learning in social networks. The research shows that social learning promotes adaptive learning for students and improves the quality of teaching in the classroom.

Social software applications have attracted the attention of educators in higher educational institutes. It is widely used in teaching to increase the interaction between learners and lecturers (Schroeder et al., 2010). For example, teachers can build online learning groups through the social network application Facebook. In the study group, students can supervise each other, and teachers can efficiently manage the learning process. Galanis et al. (2016) described a social dynamic learning framework that can promote users’ informal digital learning. Based on this system, a peer evaluation mechanism was also put forward for personalized learning recommendations. The increase in collaborative learning is also correlative to the emergence of social networks. Stantchev et al. (2015) introduced a cloud computing service into an e-learning platform that could be used to assess the user’s knowledge level by artificial intelligence mechanisms and recommend education and training for users’ personal learning and their career development. However, social networks generate massive amounts of information during user’s interactions. To address this problem, a neural collaborative filtering model was presented to predict the ratings, and the recommendation method of this model is feasible and effective.

In this paper, we have reviewed some journal articles, from the years 2014 to 2018, related to e-learning recommender systems, from which a coding scheme was proposed, and an in-depth analysis was conducted to evaluate the effectiveness of the systems.

To be specific, the following series of research questions are investigated:

RQ1.

What were the primary metrics for e-learning systems?

RQ2.

What were the primary evaluation metrics for recommendation algorithms?

RQ3.

What were the primary recommendation filtering technologies in e-learning systems?

RQ4.

What were the important phases of the recommendation process?

RQ5.

What were the learning outcomes achieved in the e-learning recommender system?

Research methodology

Data collection and processing

In this research, the Web of Science database was selected as our data source because it is one of the most influential journal article collections. In the database, 221 articles were found by using the query of “recommendation” and “e-learning.” Then, the publication period was set for the years 2014 to 2018 to ensure that our research kept up with the newest development trends of this field. As our research subject belongs to the category of educational technology, nine reputational and representative journal publications were selected as our principal research sources, such as the Journal of Computing in Higher Education. Therefore, there were a total of 31 articles in the query results. Moreover, to ensure that these articles truly match our e-learning recommender system, the following inclusion criteria were established to analyze each article. Each article must be relevant to the recommendation technology, and the research topic must be included in the area of e-learning. Specifically, the article should use one or more learning models to support users’ learning or teaching processes. Moreover, these learning systems must have some specific recommendation targets for the users as well, such as recommendations of teaching modes or learning resources. According to the criteria, 11 irrelevant articles were filtered out, leaving 20 articles that met the criteria for our research data set. For example, Yilmaz and Keser (2016) discussed the effect of reflective thinking activities in an e-learning system as well as providing suggestions for learning environment design. However, this study is not involved in the recommendation technologies and relevant content recommendations. Another example is an article about the blended learning approach in higher education based on multimedia tools (Bicen et al., 2014). Figure 3 illustrates our data collection and processing analysis process.

The assessment for e-learning recommender systems

To evaluate the e-learning recommender system, this section develops coding schemes from five research areas including learning systems, recommendation algorithms, filtering techniques, recommendation processes and learning outcomes. The details are shown as follows.

Codes for e-learning systems

On the personalized learning platform, learner’s ability is also an essential factor that we need to consider, in addition to learners’ preferences, interests or learning behaviors. However, in the era of information explosion, web-based learning systems have emerged with many problems. Therefore, in the e-learning system, as suggested by Chen et al. (2005), the following issues need to be addressed, including disorientation, cognitive overload, adaptive mechanisms and information overload.

Disorientation refers to an event that causes a learner to lose direction in hyperspace or deviate from the learning goal without knowing it (Bhatti et al., 2017), while cognitive overload indicates that “learner’s intended cognitive processing exceeds the learner’s available cognitive capacity” (Mayer and Moreno, 2003). The cognitive load theory shows that learners can effectively absorb and retain information only when they do not acquire information when in an “overloaded” mental capacity (Chandler and Sweller, 1991). To be specific, learners’ short-term memory can only retain a certain amount of data in a short period of time. If the information exceeds our available knowledge capacity, then the data cannot be retained in our brains for a long time.

Adaptive learning mechanisms could provide learning guidance according to individual differences (Huang and Shiu, 2012). For example, a system with an adaptive mechanism can be adjusted based on the course structure and learning preferences, which could adapt to the teaching styles, and provide students with suitable courses for their personal learning (Graf, 2009). Information overload is used to describe the difficulty of understanding an issue (Yang et al., 2003), and when we input too much information about the issue into the system, there will be a selective mechanism to make decisions effectively. However, due to the limited cognitive processing capacity of decision-makers, problems with inefficient decision-making may occur (Speier et al., 1999).

Codes for algorithm evaluations

The codes for the recommendation algorithms include two evaluation metrics such as accuracy and coverage. Accuracy is a small fraction of the correct recommendations for the total possible recommendations, while coverage measures the proportion of objects in the system’s search space. Coverage is related to the percentage of items and users for which the recommender system can provide predictions. Accuracy metrics consist of statistical accuracy and decision support accuracy. Statistical accuracy metrics indicate the actual user ratings, and the accuracy of the filtering technology is assessed by directly comparing the introduced ratings, while decision support accuracy metrics include the “Reversal rate, Weighted errors, Receiver Operating Characteristics, and Precision-Recall Curve, Precision, Recall, and F-measure” (Isinkaye et al., 2015).

Codes for filtering technologies

The codes for recommendation filtering technologies include content-based filtering, collaborative filtering and hybrid filtering. Content-based technology is a domain-dependent algorithm that emphasizes the analysis of the attributes of items to generate predictions, such as web pages, user profiles, publications and news.

Collaborative filtering is “a domain-independent prediction technology that cannot be easily and adequately described by metadata such as movies and music” (Isinkaye et al., 2015). Suggestions are provided by calculating the similarities between their configuration files to match users with relevant interests and preferences. Collaborative filtering includes memory-based techniques and model-based techniques. Memory-based techniques use user ratings to calculate similarities between users or items. Additionally, the items are combined with a neighbors’ performances to generate recommendations once the user’s neighbor is found. Model-based techniques, such as machine learning or data mining technologies, are usually used to develop models to predict user ratings for unrated items, which are similar to a user-based or item-based neighborhood algorithms and are more scalable when dealing with large sparse data sets (Sandvig et al., 2008).

Hybrid filtering combines many different recommended techniques for better system optimization and to avoid some limitations and problems with pure recommender system. Most commercial systems are hybrid, such as the Google news recommender system (Das et al., 2007). This technique can be further classified into seven categories including weighted hybridization, switching hybridization, cascade hybridization, mixed hybridization, feature-combination, feature-augmentation and meta-level (Isinkaye et al., 2015).

Codes for recommendation processes

The codes for the recommendation process phase consists of the information collection phase, the learning phase, and the prediction/recommendation phase (Isinkaye et al., 2015). The information collection phase will collect users’ information to generate a user profile or model for the prediction tasks, including user attributes, behaviors or the content of the resources accessed by the user. This phase includes explicit feedback, implicit feedback and hybrid feedback. Explicit feedback is explicit input of user’s interests in the item, and the implicit feedback is the indirect inference of user preferences by observing the user’s behavior, while mixed feedback is obtained by combining explicit feedback and implicit feedback. In addition, the learning phase is also an essential phase in the recommendation process. It is the application of learning algorithms to filter and utilize the feedback information collected during the information gathering phase. The last phase is called the prediction or recommendation phase, which recommends or predicts the types of items the user may like.

Codes for learning outcomes

As suggested by Fu and Hwang (2018), the main codes for learning outcomes include 15 dimensions. In this review, as the focus is the recommender system for e-learning, we adapt the original 15 dimension codes to six categories, skills, cognition, behavior, affection, correlations and others, which is more suitable to the context of the recommender systems. For affection, it can be further elaborated as eight subcategories according to Fu and Hwang (2018). The category cognition, is related to cognitive learning outcomes. The third category “skills” refers to the accuracy and fluency of the user’s operation or demonstrations on the online learning platform. The categories “behavior” and “relevance” refer to the effect relationship, learning behavior, or cause-effect relation, including valuable model, social influence, or contributing factors at different research levels.

Research results

Distribution of e-learning systems

As shown in Figure 4, approximately 80 percent of the articles (16 out of 20) mentioned the adaptive mechanism as an indicator for the e-learning system. Two articles mentioned the metric of “information overload” and each of the remaining articles involved “disorientation” and “cognitive overload,” respectively. Therefore, it can be intuitively observed that the adaptive mechanism is a critical metric for the system because it can automatically and quickly create system feedback under the same external conditions and adjust the original strategy to achieve the best optimization goals (Yu et al., 2005).

Distribution of algorithm evaluations

As shown in Figure 5, there are six articles in connection with the statistical accuracy, and four articles used the decision support accuracy to evaluate the recommended algorithm. Therefore, in all the research data sets, half of the objects were involved in the metric of accuracy. However, only two articles mentioned the coverage of the algorithm. The results showed that the recommendation accuracy is more critical to evaluating the quality of an algorithm when it was compared with the metric of coverage, as accurate recommendations can provide users with useful information and help them improve their learning or work efficiency. Furthermore, the accuracy of the algorithm is related to the user’s experience and, to a large extent, to the performance of the system.

Distribution of filtering technologies

To understand how an e-learning system provides recommendation services, we have studied the recommendation mechanism of system models from the perspective of a recommending filtering technology. As shown in Figure 6, recommendation filtering techniques include content-based filtering, collaborative filtering and hybrid filtering. Figure 6 indicates that hybrid filtering is the most commonly used technology in the e-learning platforms, while the other two technologies are also prevalent. The total frequency of usage of these three technologies is 61. This means that the learning system typically does not use one single technology but uses two or more technologies to complete the recommendation tasks.

Figure 7 shows the distribution of the two subcategories of the collaborative filtering technique. The analysis results show that the model-based technique is used more frequently, probably because it can process large sparse data sets better and faster than the memory-based approach (Sarwar et al., 2001).

Since most e-learning platforms use hybrid filtering techniques, it is essential to investigate the subcategories of this technology. Hybrid filtering techniques can be further divided into seven subcategories. As shown in Figure 8, weighted hybridization and feature-combination were used six times in our review articles, accounting for 26 percent of the total number. The feature-augmentation technique is seldom used, and only two papers use this filtering technique, accounting for 9 percent of the total. However, we have not found any articles using the switching hybridization approach.

Distribution of recommendation processes

As shown in Figure 9, the information collection phase is the most crucial phase of the recommendation process, followed by the prediction/recommendation phase. From the results in Figure 10, it could be inferred that explicit feedback is the primary and most direct way for the recommender system to gather information. Among the 20 articles we studied, seven articles used implicit feedback, six articles used hybrid feedback and the proportion of articles using these two channels to collect feedback data were basically the same.

Distribution of learning outcomes

The learning outcome is an important indicator to evaluate the quality of the learning system output. Figure 11 shows the six types of learning outcomes that may be measured in an e-learning platform. Based on our study of electronic recommender systems, affections and correlations are the most frequent learning outcomes in these studies, followed by cognition. The behavior is also one of the elements of learning outcomes that needs to be considered in the recommender systems. As the first learning outcome “affection” is a vital research factor for all categories, it was divided into eight subcategories in order to do further research. As shown in Figure 12, among the 18 studies involving affections, the technology acceptance/learning intention, opinion/learning experiences of students and learning attitudes/ expectation of learning engagement are the three most frequently measured indicators from a total of 12 items, which account for approximately 66.67 percent of the category of “affections.”

Discussion

In this section, we will analyze our review study from a macro perspective. Based on the pedagogical approaches for e-learning systems, we divided the research data set into the following eight categories, namely, game learning, collaborative learning, social learning, adaptive learning, blended learning, lifelong learning, flipped learning and others (relevant to semantic web).

As shown in Figure 13, collaborative learning and social learning are often selected as research objects for learning recommender systems. Among 20 review articles, 60 percent (12 out of 20) articles were related to these study subjects. Second, there are four articles about the study of semantic web technology, which accounts for approximately 15.38 percent of the total number of pedagogical methods. In the remaining articles, three articles discussed the educational model of game learning, and there is only one paper referring to flipped learning. For example, Hsieh et al. (2017) proposed an adaptive filtering mechanism by using the LINE as a tool for innovative teaching in the flipped learning environment.

As shown in Table I, the articles published between the years 2014 and 2018 have mentioned social learning, which indicates this research topic has been important for online recommender systems in recent years. However, adaptive learning, blended learning and lifelong learning have been less involved in the recommender systems of recent years. Therefore, it could be simply predicted that game learning, collaborative learning and social learning will be the research trends in future online learning systems. Chang and Hwang (2019) summarized the number of papers published from 2007 to 2016 about mobile game-based learning and the statistics show the increasing trend of study on game learning. As collaborative learning is a development trend in the twenty-first century, Efendi and Yulastri (2019) used the collaborative learning model to demonstrate its effectiveness on computer network courses, while Khechine and Augier (2019) conducted an investigation to study the factors affecting the adoption of the social learning platform and its conclusion indicates that social learning has great potential. According to the latest paper mentioned above, the study of game learning, collaborative learning, and social learning have become hot research topics which will attract more scholars’ attention in the next few years.

Conclusion

In this systematic review study, some research questions have been answered by involving metrics for e-learning systems, evaluation metrics for recommendation algorithms, recommendation filtering technologies, phases of recommendation processes and learning outcomes. Further, we have developed a coding scheme for the evaluation of each selected journal article, performed a statistical analysis on each coding category, and compared the differences of the elements under the same evaluation index. We then summarized the pedagogical approaches of each research article, discussed the distribution of the research topics, and predicted three major development trends for e-learning systems in the future.

After an in-depth analysis of the research studies, we have reached the following conclusions. Between the years 2014 and 2018, 80 percent of the review articles adopted the adaptive mechanism as one metric of the learning system. Because it is critical to maintain the system stability and performance optimization of the e-learning platform with self-adjustments and feedback mechanisms, in regard to the evaluation metrics of the recommendation algorithm, approximately 50 percent of the research literature will give priority to the accuracy of the algorithm, while only 10 percent of the research articles mention the coverage metric. In terms of the recommendation filtering technology, nearly 38 percent (23 out of 61) of the articles apply the hybrid filtering technology, but both content-based filtering and collaborative filtering are equally important to improve the service quality of the learning recommender system. The information collection phase is an essential part of the recommendation process phase, including explicit feedback and implicit feedback. The explicit feedback indicates some information that could be directly obtained, such as users’ preference, while implicit feedback indirectly predicts how much interest the user has for the item based on user behavior. With these feedbacks, the system can accurately recommend preferred information or content to the user. Finally, affections and correlations are the most prominent among the six main types of learning outcomes, which account for 60 percent (35 out of 58) of all possible outcomes.

In summary, this paper reviewed the research on recommender systems for e-learning from the years 2014 to 2018 and explored the future direction of the educational research field. Nevertheless, some potential new developments and applications of educational technologies are presented, such as collaborative tagging techniques, semantic web technologies, and so on.

Figures

Figure 1

The distribution of the number of articles in e-learning recommendation systems from the years 2014 to 2018

Figure 2

The distribution of the citation frequency of 31 articles in the field of e-learning recommender systems from the years 2014 to 2018

Figure 3

The process of data collection and processing

Figure 4

The distribution of the metrics for e-learning recommender systems

Figure 5

The distribution of evaluation metrics for recommendation algorithms

Figure 6

The distribution of recommendation filtering techniques for research data set

Figure 7

The distribution of the two subcategories of collaborative filtering

Figure 8

The distribution of the seven subcategories of hybrid filtering

Figure 9

The distribution of the phase of the recommendation process

Figure 10

The distribution of the three subcategories of the information collection phase

Figure 11

The distribution of the learning outcomes of the e-learning recommender system

Figure 12

The distribution of the eight subcategories of affection

Figure 13

The distribution of pedagogical approaches of the research studies in e-learning recommender systems

Table I

The statistics of the research directions of the pedagogical approaches from the years 2014 to 2018

	2014	2015	2016	2017	2018
Game learning			✓	✓	✓
Collaborative learning		✓	✓	✓	✓
Social learning	✓	✓	✓	✓	✓
Adaptive learning	✓
Blended learning	✓	✓
Lifelong learning	✓	✓
Flipped learning				✓
Others (Semantic Web)	✓	✓		✓	✓