Literature review as a key step in research processes: case study of MA dissertations written on EFL of Saudi context

Abstract

Purpose

The aim of this study is to find out the most common types of literature review and the accuracy of citing information related to topic in question among Saudi English as a Foreign Language (EFL) postgraduate students at Al-Baha University. This study also aims at revealing the quality of the literature review written by researchers.

Design/methodology/approach

This qualitative study used content analysis to investigate 15 unpublished Master of Arts (MA) dissertations written on EFL of Saudi context. They were analyzed qualitatively using criteria modified from Snyder's (2019) model which is considered a potential method for making theoretical and practical contributions of literature review.

Findings

The findings of the study showed that students favored the systematic review over the integrative. Additionally, data showed that students were lacking in paraphrasing and organizing cited information coherently and appropriately. Moreover, students' performance was better in design, conduct, and data abstraction and analysis criterion, whereas they seemed rather weak in structuring and writing the review criteria.

Originality/value

The significance of the study is to provide researchers with methodological guidance and reference to write a comprehensive and appropriate literature review. Based on the findings, this study concluded with some implications that aim to assist researchers in carrying out their studies professionally. Furthermore, the findings provide decision-makers in higher education institutions with important practical implications. In light of the study's findings, it is suggested to carry out further research investigating postgraduate students to find out their perceptions and attitudes regarding the quality standards of scientific research writing and the paraphrasing strategies.

Citation

Alsalami, A.I. (2022), "Literature review as a key step in research processes: case study of MA dissertations written on EFL of Saudi context", Saudi Journal of Language Studies, Vol. 2 No. 3, pp. 153-169. https://doi.org/10.1108/SJLS-04-2022-0044

Introduction

The way a researcher is building his/her research and linking it to current knowledge is like building a block of academic research activity, no matter which discipline it relates to, thus it is a priority step (Snyder, 2019). In a definition by Liberati et al. (2009) cited in Snyder (2019) “A systematic review can be explained as a research method and process for identifying and critically appraising relevant research, as well as for collecting and analyzing data from said research.” (p. 334). A literature review is an important part of any research as it is considered a foundation of the type of research.

As in Snyder (2019), literature review is a written text of a published study that includes current knowledge and up-to-date information about the latest findings of science on a particular topic, including substantial discoveries as well as theoretical and practical contributions from scholarly research groups. A literature review as defined by Hart (1998) (Cronin et al., 2008 cited in Ramdhani et al., 2014) “is an objective and thorough summary and critical analysis of the relevant, available research and non-research literature on the topic being studied” (p. 48). A literature review requires a compound series of abilities to learn topics to explore and acquire and retrieve literature searching skills. Additionally, it requires the ability to develop, analyze and synthesize data to be keen on reporting and writing normally at a limited scale of time. Scholars divided the literature review into two types. The first is “Traditional or Narrative Literature Review”. This type of review criticizes and summarizes the body of the literature to draw conclusions about the topic under consideration. The basic aim of this review is to support a reviewer with a complete review to understand the knowledge and to show the implication of new inquiries. The second one is “Systematic Literature Review” which reviews the literature in a specific subject area to employ a more rigorous and well-defined approach. A systematic literature review is often used to solve a specific medical practice question (Parahoo, 2006; Davis et al., 2014; Almelhes, 2020). Some studies regard “meta-analysis” as a type of systematic review, which is primarily a statistical method that entails evaluating the research results among many studies on the very same topic using standardized statistical tests in drawing conclusions and identify patterns and trends between research results (Polit and Beck, 2006; Dundar and Fleeman, 2017; Almelhes, 2020).

A review of the literature is the core systematic investigation and therefore can help optimize research questions by identifying knowledge gaps. Correspondingly, it can assist to motivate new research innovations while also increasing one's understanding of a subject. A literature review can help a beginner researcher delve deeper into appropriate models for future research (Antons and Breidbach, 2018), and provide description of the data collection and analysis tools. Where and when the literature review is conducted is frequently dictated by whichever strategy is qualitative or quantitative. Based on the purpose of conducting the evaluation and the key categories and purpose of the research, different types of literature reviews could be used. Knowing how to write a literature review is a professional discipline that is necessary for practice. For several students, starting with literature is a difficult challenge. Master of Arts (MA) students seem to have less experience compared to doctoral ones and professional scholars, so we can find them tend to use thoughtful, thorough and critical literature since there are few guides at the master's level (Kamenstein, 2017). Consistently, Randolph (2009) states that literature written for MA theses is less comprehensive and exhaustive than in doctoral dissertations. Just about all students may be confused about where to begin and how to choose a topic, how so many publications they will need, what a literature review entails. This state of despair sometimes appears clearly with postgraduate students, mainly English as a Foreign Language (EFL) MA students and is the main obstacle for many students. Therefore, the purpose of this paper is to find out:

1. The most common types of literature review among Saudi EFL postgraduate students in Al-Baha University,

2. The accuracy of citing information related to the topic in question and

3. The quality of the literature review written by the MA researchers at Al-Baha University.

Therefore, the study raises three questions:

1. What are the most common types of literature review among Saudi EFL students at Al-Baha University?

2. To what extent is the accuracy of citing information related to the topic in question?

3. To what extent is the quality of the literature review written by the MA researchers at Al-Baha University?

Steps and phases of doing a literature review

Designing the review

Why should this literature be reviewed? Do we really need literature in this area of our topic? And what literature review would be the great type for contribution? Indeed, these questions would be better borne in the mind of a researcher before starting to review the literature because they determine the likelihood of the review and the impact it might have on the research community (Antons and Breidbach, 2018). As it is a hard work to conduct a literature review, the topic must interest both author and the reader. Hence, first of all, it is better to scan the points to relate to existing knowledge. Moreover, Palmatier et al. (2018) stated that any criterion related to the on-focus topic should be directed by the research questions.

Conducting the review

Conducting a review is required after deciding on the purpose, questions and type of approach that better suits the topic in question. Additionally, it is better to appropriately test the review process and protocol before performing the main review. To ensure the quality and reliability of the search protocol, it is important to use two reviewers to select articles depending on the nature and scope of the review (Antons and Breidbach, 2018; Almelhes, 2020).

Analyzing the literature

To conduct appropriate analysis, it is important to consider how the articles will be used. Meanwhile, abstracting information needs to be professionally measured (Palmatier et al., 2018). They can be put into descriptive information (e.g. authors, year of publication, topic or type of study), or in effects and findings format, conceptualizations or theoretical perspective. Additionally, it is better to avoid any differences in coding and monitoring the data abstraction carefully during the review process in order to ensure quality and reliability. Researchers should ensure that their literature is appropriate to answer the selected research question.

Writing a review

The final review of any article depends on an approach that requires types of different information and different levels of details like standards and guidelines that explicitly address how literature reviews should be reported and structured (see Table 1, below). Standards and guidelines for systematic narrative reviews (Wong et al., 2013) or guidelines for integrative reviews (Torraco, 2005) should be considered in the final review too. Moreover, how literature was identified, analyzed, synthesized and reported by a researcher is necessary to describe transparently the process of designing the review literature. Literature reviews can result in a historical analysis of the development within a research field (Carlborg et al., 2014; Almelhes, 2020) or can be any agendas for further research (McColl-Kennedy et al., 2017), besides, conceptual model or categorization (Snyder et al., 2016; Witell et al., 2016), or can be evidence of an effect (Verlegh and Steenkamp, 1999).

The process of undertaking a literature review

Regardless of the method used to carry out the literature review, there seems to be a myriad of activities to be carried out and decisions made in order to build an assessment that satisfies the criteria for publication (for specific considerations with regards to each phase, as seen in Table 2). There are four phases that demonstrate and discuss the essential decisions and questions associated with conducting a literature review (as in Table 2 below): (1) designing the review, (2) conducting the review, (3) analysis and (4) structuring and writing the review (Snyder, 2019). This procedure arose from real-world and practical experience and is a synthesis of and impact by diverse rules and specifications for literature reviews (e.g. Liberati et al., 2009; Tranfield et al., 2003; Wong et al., 2013).

Types of literature review

Systematic literature review

As described by Davis et al., (2014) and later by Dundar and Fleeman (2017), systematic reviews have first developed within medical sciences to synthesize research findings in a systematic, transparent and reproducible way. It can be a process for identifying and critically appraising relevant research for collecting and analyzing data from previous studies (Liberati et al., 2009; Almelhes, 2020). It aimed at identifying all empirical evidence that fits the pre-specified inclusion criteria to answer a particular research question or hypothesis. Bias can be minimized to provide reliable findings from conclusions and decisions (Liberati et al., 2009). Often statistical approaches are used to integrate the results of the topics in question. It combines results from different studies to evaluate and compare and identify patterns, disagreements or relationships (Davis et al., 2014) to assess them. It can be used to determine the continuity of effects across studies and to discover types of future studies that are required to be conducted to demonstrate the effect. Besides, techniques were used to discover which study-level or sample characteristics affect the phenomenon (Davis et al., 2014). The primary goal of a systematic review is to provide as comprehensive a list as possible of all studies whether published or unpublished, and these studies concerning a specific subject (Ryan et al., 2007; Dundar and Fleeman, 2017).

A systematic review needs to use standards as a roadmap for collecting studies (Livinski et al., 2015). Systematic review design has covered the following criteria: (1) studies related to students' attitudes; (2) the engagement of the learning process and (3) the outcomes of studies regarding speaking, writing and reading skills (Antons and Breidbach, 2018; Almelhes, 2020).

Semi-systematic review

The semi-systematic or narrative review approach hinders a full systematic review process. It is designed for different conceptualized and various studies that were studied by groups of researchers within various disciplines (Wong et al., 2013; Dundar and Fleeman, 2017).

Since it is hard to review every single article relevant to the topic, a different strategy must be developed (McColl-Kennedy et al., 2017). It aims at overviewing a topic and how research has progressed over time and developed. Generally, it seeks to identify and understand all potentially relevant research traditions and synthesize them by measuring effect size (Wong et al., 2013) and provides a considerate understanding of complex areas. It is potentially contributed to a useful analysis for detecting themes, and theoretical viewpoints of specific research disciplines as well as to identifying components of a theoretical concept (Ward et al., 2009). Thus, gain the ability to map a field of research, synthesize the state of knowledge and create an agenda for further research or the ability to provide a historical overview of a specific topic.

Integrative review

An integrative review is closely related to the semi-structured (integrative or critical review) approach. Usually, it has a different purpose from the semi-structured review which aims to assess, critique, and synthesize the literature in a way to develop new theoretical frameworks and perspectives (Torraco, 2005). Generally, integrative literature reviews are intended to address mature or new topics. Additionally, seek to emerge topics to overview the knowledge base, critically review and potentially reconceptualize and expand on the theoretical foundation of the specific topic. It requires a more creative collection of data (Whittemore and Knafl, 2005). A review of good literature does not summarize the sources, but rather analyzes, collects and evaluates them accurately to form a clear and general picture of existing knowledge or science on this topic.

Text borrowing skills

Text borrowing and incorporating other people's written ideas into one's own scholarly work are useful qualities to have in the world of academia, particularly for those pursuing higher education. Text borrowing expertise widely used in academic writing includes direct quoting, paraphrasing and summarizing. When contrasted to paraphrasing, directly quoting from the primary material is far more feasible, easier and less complex. There is really nothing inappropriate with integrating quotations; nevertheless, as Davis and Beaumont (2007) point out, overusing quotations does not really represent highly proficient writing. Rather, academic writing motivates the use of paraphrasing, drawing conclusions or synthesizing skill sets.

Paraphrasing is described as reiterating a statement in such a manner that both sentences are lexically and syntactically distinct whilst also remaining semantically equivalent (Amoroso, 2007; Davis and Beaumont, 2007; McCarthy et al., 2009). At least two echoes are implied by this description: reading process skills and writing ability. As a result, according to McCarthy et al. (2009), paraphrasing is often used to aid comprehension, enhance previous knowledge and assist the development of writing skills.

According to cognitive psychology literature, paraphrasing is mentally demanding. As the content to be paraphrased has become more complicated, students are more likely to use simplified processing, resulting in patchwork written text (Marsh, Landau and Hick in Walker, 2008). Walker adds that just imagining about paraphrasing takes a substantial amount of cognitive vitality, and when the physical writing process starts, individuals have restricted opportunity to undertake thoughtful, systematic processing to ascertain if they paraphrased correctly. These complicated characteristics of paraphrasing cause some challenges. In the Japanese context, Iwasaki (1999) discovered four major areas of difficulty: varying behavioral patterns of parts of speech, subject limitation, context-specific paraphrasing and “blank” locating. There seems to be little proof, and data obtained from extensive research dedicated to examining paraphrasing-related concerns in the Indonesian context. Despite an abundance of survey participants, Kusumasondjaja's (2010) survey did not test students' paraphrasing abilities. It appears that paraphrasing is not represented, is described vaguely or is purely regarded as changing the existing source without stating the extent of adjustment.

In the Saudi context, Alaofi (2020) investigated the key problems that Saudi graduate students usually face when summarizing and paraphrasing source texts in EFL. Nine Saudi students attending university degrees in multiple fields were questioned using a qualitative approach. The study's findings revealed that a variety of barriers may exacerbate students' challenges with the skills under examination. These were students' insufficient English proficiency is the first root of complexities in summarizing and paraphrasing original text, followed by issues with students' writing styles and, finally, poor reading comprehension skillsets.

Methodology

As mentioned above, the purpose of this study is to analyze and synthesize findings from the content that is written in the literature review section of 15 unpublished MA studies. These studies were written in the Saudi context and conducted by MA postgraduate students of Al-Baha University. Additionally, to find out the most common types of literature review used by Saudi EFL postgraduate students in Al-Baha University, and to measure the accuracy of citing information related to topics in question, besides finding out the criteria and assess the quality of literature review written by MA researchers, to come out with rich findings that can guide undergraduate students in writing and reviewing knowledge related to their theses and research papers. Additionally, it can help postgraduates and other academic researchers to build a tidy content of literature and coherent procedures for research writing. Thus, this research is done qualitatively using content analysis taking into account the discipline, type of literature review, and contribution to see how successfully these researchers attract readers' attention and satisfy their needs, and in the long run, increase the quality of research and to develop better and more accurate hypotheses and questions.

Procedures

To measure the research questions, 15 MA dissertations were selected randomly and carefully analyzed accordingly. The analysis of these 15 studies focused mainly on finding out the common types of literature review used by Saudi EFL students in Al-Baha University, and finding the accuracy of citing information, besides assessing the quality of the literature review of the selected MA research. Synder's (2019) model for assessing the quality of literature review is used as a criterion to analyze these MA studies. All are written in the field of English Language Teaching (ELT) settings. Therefore, it will be a potential step in making theoretical and practical contributions to literature review as a method to clarify what a literature review is, how it can be used and what criteria should be used to evaluate its quality. Thus, in this paper, the contribution differentiates between several types of literature review methodologies such as systematic, semi-systematic, and traditional/integrative approaches and how the procedures and the quality were shown (see Appendix). Besides, presenting real practices that may be met when reviewing literature in EFL research. Additionally, it provides context and guidance to students and academics to use the literature review as a method to synthesize their research in question.

As in Appendix, the criteria used contained four phases: (1) design (includes 6 dimensions); (2) conduct (includes 5 dimensions); (3) data abstraction and analysis (includes 5 dimensions); and (4) structuring and writing the review (includes 5 dimensions). To show that the criterion has been met, the researcher used the symbol (√) as an indication system or vice versa (×) if it was not. The 15 kinds of research were coded using numbers (i.e. each research was given a number from 1 to 15). Then each research was checked according to the dimension of each criterion of each quality. These 15 unpublished MA studies were collected from the College of Arts and Humanities in Al Aqiq main campus, where the postgraduate dissertations were archived, and these studies were conducted during the period from 2013 to 2018. The reason for not selecting newer studies after 2018 is that this paid master's program has been discontinued and has resumed in mid-2021. To ensure the quality of the assessment and the analysis according to Synder's standard, the researcher got help from jury members of three PhD holders (voluntarily) who work in the Department of English at the College of Science and Arts in Qilwah. They had more than ten years of experience in the field of teaching and scientific research. The research took place in a round-held table for a number of meetings and asked them to review and evaluate the MA research according to Synder's criteria. The evaluation continued for three months, and each phase and its dimensions were discussed in separate sessions. The evaluation and discussion took place during the first term of the academic year 2021. Step by step the researcher continuously discussed with the jury members their evaluation (see Appendix).

Analysis

The analysis section was divided into two parts. The first part displayed the data gathered to measure the first and the second questions, whereas the second part displayed the third question.

Discovering common types of literature review and evaluating the accuracy of citing information

Part 1: The main types of literature review (traditional or narrative, systematic, meta-analysis and meta-synthesis) were scrutinized and analyzed in light of their qualities and procedures. In this paper, three types are chosen to be judged accordingly. They are systematic, semi-systematic and traditional/integrative approaches. As mentioned above, the 15 MA projects were handed over to the reviewers (the researcher's colleagues). After long and regular sessions, they concluded their results to the researcher. They revealed that studies 1, 10 and 15 showed a masterpiece reflection of the systematic review approach. In this sense, these studies synthesized and compared evidence between the two studies. Another example is that these studies in the introduction section produced a clear and rationale connection between the topic and literature written in the same field of the study. These studies also showed that the information provided is reliable and based on proven facts. Additionally, the information is verified against other reliable sources. To be more realistic, we must evaluate all sources before deciding whether to incorporate what was found into the literature review (Synder, 2019). Moreover, resources need to be evaluated to make sure that they contain information, which is valuable and pertinent, in this point, this study is consistent with what was found by other researchers (Liberati et al., 2009; Tranfield et al., 2003; Wong et al., 2013; Synder, 2019). These studies presented a rich literature that is displayed in various types of periodicals that include scholarly journals of high impact factors and intensive readability.

Generally, studies (1, 3, 4, 6, 8, 10, 11, 12, 13, 14 and 15) used systematic reviews to answer their highly structured and specific research questions. They undertook a more rigorous approach in reviewing the literature they presented in their research. In contrast, a traditional or narrative literature review usually adopts a critical approach in a way that analyzes and summarizes to address intensive information to shed light on new ideas, bridge gaps or/and cover weaknesses of previous research, studies endeavored a very high and accurate criterion of evaluating literature review Ryan et al. (2007). On the other hand, studies (2, 5, 7 and 9) used both semi-systematic and integrative/traditional literature reviews. This clearly showed that the research questions were broad. On the other hand, in the introduction section, these studies used a semi-systematic literature review. For accuracy purposes, these studies presented a piece of reliable information. All the information displayed in these studies was error-free. Additionally, it is easy to say that the information shown was based on proven facts and can be verified against other reliable sources. All that cited in these studies in the literature review section was taken from famous and well-known periodicals. They can be completely described as facts shown without any bias. When looking back to what the researchers presented, it is easy to see that information presented was currently published to show the currency matter of the researchers' topics. The coverage of information has met in-depth the information needed to build up a literature review process. Accordingly, the researchers reviewed rich and accurate literature written about the focus topics to rationalize their objectives in conducting their research. All the information shown by researchers was presented without any bias. Thus, each study presented more than four references to show the accurateness of the literature. Additionally, the information presented is highly met and covered the needed information, and provided a basic and in-depth coverage. To meet the aims of systematic review (as in Dundar and Fleeman, 2017), to some extent, these studies provided a complete list of all possible published and unpublished studies relating to the researcher's subject matter.

To deal with the accuracy of citing information, in studies (2, 3, 4, 5, 9, 10, 11, 12, 14 and 15) one can find that, from the beginning, in the introduction section, these studies started citing from very recently published researches. Most interestingly, the researchers used the paraphrasing method to cite information related to the researchers' topic. However, they did not paraphrase appropriately. These studies used a paraphrasing method to make the information cited more reliable, error-free, based on proven facts and can be verified against other reliable sources. In this sense, these studies presented information that can be more accurate. But unexpectedly, those who used the paraphrasing method lacked professionality in treating the original text and formulating it using their own words (i.e. there is an apparent weakness in meaning between the original source and the paraphrased text). Again, the studies showed that researchers intended the purpose of the information in a precise way initiating that in the introduction. The studies also presented facts that were proven by famous writers.

Assessing the quality of a literature review

Part 2: As empirical research, literature reviews need assessment and evaluation (Palmatier et al., 2018). The literature review quality must have both depth and rigor to determine a suitable strategy for choosing topics and apprehending data and insights and to recite previous studies slightly. The quality of the literature review needs to be replicable to make the reader easily replicate the topic and reaches similar findings. Additionally, they must be useful for scholars and practitioners. Normally, the evaluation of different types of literature reviews is considered to be challenging. However, some guidelines could be used as a starting point to help researchers in evaluating literature reviews, to examine and to assess the review criteria for rigor and depth. To assess the literature review quality related to MA studies (the 15 selected samples) in question, Snyder (2019, p. 338) suggested some guidelines as seen in Appendix.

Therefore, to find out a suitable step-by-step approach that can guide students and academics in undertaking a valid, comprehensive and helpful literature review, appropriate literature reviews have been gathered for this inquiry to investigate the criteria and quality of literature presented by selected MA studies (see Appendix). Depending on the purpose of the topic, various literature studies may be highly useful to suggest which strategy may suit the analysis and synthesis stages that greatly help in the selection and writing of the literature review on EFL context, mainly Saudi context. Thus, the selected reviews were scrutinized and analyzed considering their qualities and procedures. In this paper, only 15 MA studies were chosen to be measured accordingly.

As in Appendix, the quality of these projects was checked and graded by the reviewers (the researchers' colleagues). Looking at the reviewers' evaluation of these dissertations, one can easily find that only one out of the 15 showed complete performance and was valuable in all criteria. Meanwhile, the others showed strength in the first and second dimensions of the design criteria. However, their performance in the other parameters deteriorated greatly. In total, seven out of them performed moderately in design criteria (i.e. they achieved in dimensions, 1, 2 and 6, whereas they failed in the other 3, 4 and 5); as the other seven researchers performed in four dimensions moderately. Concerning the conduct criteria, they did very well. Therefore, ten out of the fifteen researchers fulfilled perfectly in all dimensions, however, only five performed somehow moderately in only four out of the five dimensions in the conduct criteria. Additionally, seven out of the fifteen achieved in all dimensions in data abstraction and analysis, whereas three contented in only four dimensions, meanwhile other three of them fulfilled only three, while the other two achieved two dimensions. Although researchers did very well in design, conduct, and data abstraction and analysis criterion, they seemed very weak in structuring and writing the review. They only achieved in the fourth and fifth dimensions of this criteria. However, they got zero achievements in the first dimension. Additionally, only six out of the fifteen researchers fulfilled the third dimension, whereas only five achieved the second one.

Findings and discussion

As mentioned above, the purpose of this study is to find out the common types of literature review used by Saudi EFL postgraduate students in Al-Baha University; and to find out the accuracy of citing information related to the topics in question; as well as to assess the quality of the literature review written by selected researchers. From the analysis, it was found that most researchers (11 out of 15) used a systematic review of the literature. Systematic reviews are the thorough and openly transparent type of literature review. Moreover, the most reliable and comprehensive statement about what works is that systematic reviews embark on identification, synthesis and assessment of the available proof, or qualitative and/or quantitative, as a way of generating a well-researched, and empirically derived answer to a specified research question (Petrosino et al., cited in van der Knaap, 2008). The analysis also showed that most researchers utilized paraphrasing in their citing information. Even though it eases work for the researcher, paraphrasing may poorly present information if not used well. As advised, it is better to understand the readings and put them in your own words to preserve the accuracy of the information. Thus, understanding information and then properly paraphrasing will make the work look more original and refined. In this study, the researchers in many parts of the research failed to do an accurate performance in paraphrasing (i.e. there is an apparent weakness in meaning between the original source and the paraphrased text). This demonstrated that there was no accuracy in the paraphrasing used to cite information on the topics in question.

Regarding the quality of the literature review written by selected researchers, the data were gathered using a checklist by the evaluators who voluntarily distributed it into the inquiry to help the researcher to rate the performance of the 15 MA students in their dissertations. Generally, the results showed that students (the MA researchers) were keen on the conduct phase and then to some extent on data abstraction and analysis. Therefore, they know how to conduct their research, especially they have proper measures to ensure quality data abstraction. Moreover, chose data analysis techniques appropriately concerning the overall research questions and the data abstracted. Thus, they accurately search the process for types of reviews, and they did the inclusion and exclusion processes of articles transparent which makes their sample appropriate and in concordance with the overall purpose of the review. Additionally, in the design, concerning the relationship to the overall research field, their literature review was needed, and it makes a substantial, practical and theoretical contribution; the motivation, the purpose and the research questions were clearly stated and motivated; the methodology and the search strategy were clearly and transparently described. On the other hand, they were weak in phase 1 (design), item 3 “Does the review account for the previous literature review and other relevant literature?”, and they did not clearly state the approaches for the literature review. Finally, the study found that they showed a weak achievement in structuring and writing the review, especially they did not organize articles coherently about the overall approach and research question, and the overall method of conducting the literature review was not sufficiently described, thus their studies could not be replicated.

Therefore, as shown in the purpose of this research, the review of intellectual production is most often the introduction to various theses or peer-review research articles, before presenting the methods and results, and its use is common in most academic research. Thus, the literature review represents one of the important parts of the scientific research plan (Baumeister and Leary, 1997; Torraco, 2005). It is the second part that is related to the theoretical framework of the presented research methodology. Meanwhile, it is directly and closely related to the topic. Additionally, it represents an information-rich ground for those who have the desire to know all aspects of the problem or hypothesis in question. It consumes time and requires strong analytical skills from the researchers to make a great contribution successfully as mentioned by other scholars (Boyd and Solarino, 2016; Mazumdar et al., 2005, pp. 84–102; Rodell et al., 2016). As in the study questions, and to rationalize the topic, the findings concluded in this paper showed that the analysis and criticism of the literature review may require personal experiences, and others depend on the methodological foundations. The analysis and the criticism should include various dimensions (content, methodology, the sample, reliability and results). This was evident in the performance of these students in analyzing, criticizing and citing the previous studies they refer to. Thus, a researcher should have appropriate insight and wisdom to comment on previous studies and critique them constructively through compelling scientific evidence as well as to be objective and distant from any internal ideologies or personal bias. Therefore, some ideas and techniques that contribute to the process of editing, analyzing and criticizing literature review must be known by researchers (MacInnis, 2011). Additionally, more attention should be given to structuring and writing the review mainly the organization of the review in relation to the overall approach and research questions.

Implications

The study came up with some implications that can help researchers in conducting their studies skillfully. These implications were drawn from the study's findings which may be very important for practice or conducting a literature review.

First: How to criticize the literature?

When looking at the literature review, one should focus on five main points that a study can follow. They are (1) content, (2) methodology, (3) the study sample, (4) credibility and (5) results. Content criticism: in this case, the researcher must express his/her point of view that the content of the previous studies does not include the technical framework that must be followed, and in that case, the study loses the advantage of comprehensiveness and moves away from objectivity in the way it is refuted.

Criticism related to methodology

Here, the researcher must clarify the negative and positive points in the scientific method followed in previous studies, and it is not a requirement that the literature might be negative or positive in its entirety. Accordingly, this is subject to the researcher's opinion, which is an expression of his/her point of view, and he/she has to present this according to convincing evidence which varies from one researcher to another.

Criticism related to the study sample

The researcher must mention any deficiencies in the sample under study which may be ineffective in judging previous literature, and it was possible to increase the sample size. To clarify a matter related to the research problem, the sample may not be represented in an appropriate statistical way, etc.

Credibility criticism

The researcher must verify the reliability of previous studies, and the method of ascertaining. This differs according to the methodology followed by the review studies (i.e. there is the descriptive, experimental and historical approach. For example, the historical method is distinguished by its credibility from others, and the researcher must refute that matter and follow the precise criteria in judging that, etc.). To judge the reliability of literature, the researcher must be familiar with all scientific research methods, their advantages and disadvantages, and the research hypotheses and theories that are compatible with those approaches.

Results criticism

The researcher may disagree with the results shown in previous studies. Because there is an error in the method of analyzing and presenting the data, so, the researcher must clarify the comparison between his/her findings and what was presented in other studies and indicate the extent of objectivity in each of them (Snyder et al., 2016; Verlegh and Steenkamp, 1999; Witell et al., 2016). Additionally, the researcher should address only the previous studies related to the research topic, and the link must be clear to the reader, so it makes no sense to refer to previous research or studies that do not touch the research problem from near or far.

Second: How to comment on literature?

Commenting on literature is very important. Therefore, when a researcher begins writing previous studies, it is preferable not to be satisfied with the process of summarizing them, as the main goal is to discover gaps between his research and previous studies. This matter is of great importance for the development of research ideas all along with his/her research parts (Antons and Breidbach, 2018; Witell et al., 2016). Before the researcher collects the information and data for the research plan, he/she should establish in his/her mind the necessity of achieving interconnectedness with previous studies. Additionally, the results may conclude through the analysis process, and then the method of addressing similarities or contradictions, if any. After collecting information or data of different types, whether qualitative or quantitative, the stage of clarifying the essential differences between what he reached and previous studies come as well as clarifying the implications of the research presented from the benefit of humanity in general, and as found by Palmatier et al. (2018). This must also be clearly shown in the refutation and analysis of the results. The researcher must focus on the process of reasoning, and not be satisfied with the comparison only between his/her findings and what has been found by other researchers. Thus, the benefit is that reasoning or interpretation regarding similar or contradictory results is a necessity that must be linked to the comparison process (Boyd and Solarino, 2016; Rodell et al., 2016). Thus, the importance of citing previous studies, as presented by Boyd and Solarino (2016); Rodell et al., (2016), should include:

1. Previous studies help clarify the theoretical foundations of the subject of the research to be carried out by the researcher.

2. They save time and effort for the researcher by choosing the framework for the topic of the research plan.

3. They are a wake-up call for the researcher when writing a paper by defining a method that would avoid the researcher making mistakes made by previous researchers.

4. Present the correct methodological approach to the topic of research in general.

5. They give the researcher an exemplary method to extract recommendations, findings and other proposals related to the research.

6. Literature helps the researcher in identifying references for his/her research and facilitates the process of writing.

7. They have an important role in the researcher's comparison process between the research he provides and those studies and sources.

Conclusion

As many EFL MA researchers find it difficult to choose and handle a suitable literature review that approves their writing quality, thus, this study was conducted to find out a suitable step-by-step approach that can help to undertake a valid, comprehensive and helpful literature review. In conclusion, EFL MA researchers need to search for the quality and trustworthiness of their reviews to build a rich and adequate literature review. As found in this paper, it is seen that most MA Saudi researchers favored using the systematic review rather than the integrative type. More or less, they try to avoid comparing the review rather than identifying and synthesizing, as it may seem a more complicated process. Obviously, reviewing any article that could be relevant to the topic is not a simple task; therefore, a different strategy must be developed and used carefully to fulfill the quality of literature review along with the topic in question. More interestingly and generally, the researcher found that a semi-systematic review method often possesses similarities to approaches used in qualitative research (Dundar and Fleeman, 2017), but it can also be combined with a statistical meta-analysis approach. Due to the integrative approach liability to yield a creative collection of data, it is widely used to combine perspectives and insights from previous research. Thus, the integrative approach seems to be the best method that can be used in the field of EFL because its purpose is to compare and combine rather than cover all related topics. Additionally, as the study found that students did not accurately paraphrase/summarize appropriately from other sources, additional sessions impeding paraphrasing procedures and processes will have valuable benefits and will make students better at writing research in the future. Concerning the phases of the quality of conducting research, it is important to ensure the proper measurement that qualifies the quality of data abstraction and analysis techniques that deal with the overall research questions accurately. Furthermore, searching for proper types of reviews, article transparency and the appropriate sample should fit the purpose of the review. Finally, among the broader implications of the study, it is expected that the construction of master's programs (courses path) should be reviewed, and focus should be given to teach students the quality standards of research writing and how to analyze and critique them in a better way.

Table 1

Approaches to literature reviews

Approach	Systematic	Semi-systematic	Integrative
Typical purpose	Synthesize and compare evidence	Overview research area and track development over time	Critique and synthesize
Research questions	Specific	Broad	Narrow or broad
Search strategy	Systematic	May or may not be systematic	Usually not systematic
Sample characteristics	Quantitative articles	Research articles	Research articles, books, and other published texts
Analysis and evaluation	Quantitative	Qualitative/quantitative	Qualitative
Examples of contribution	Evidence of effect	State of knowledge	Taxonomy or classification
	Inform policy and practice	Themes in literature	Theoretical model or framework
		Historical overview
		Research agenda
		Theoretical model

Note(s): Adopted from Snyder's (2019, p. 334)

Table 2

Important questions to consider in each step of the review

Phase 1: Design

Is this review needed and what is the contribution of conducting this review? What is the potential audience of this review? What is the specific purpose and research question(s) this review will be addressing? What is an appropriate method to use for this review's specific purpose? What is the search strategy for this specific review? (Including, search terms, databases, inclusion and exclusion criteria, etc.)

Phase 2: Conduct

Does the search plan developed in phase one work to produce an appropriate sample, or does it need adjustment?

What is the practical plan for selecting articles?

How will the search process and selection be documented?

How will the quality of the search process and selection be assessed?

Phase 3: Analysis

What type of information needs to be abstracted to fulfill the purpose of the specific review?

What type of information is needed to conduct the specific analysis?

How will reviewers be trained to ensure the quality of this process?

How will this process be documented and reported?

Phase 4: Structuring and writing the review

Are the motivation and the need for this review clearly communicated?

What standards of reporting are appropriate for this specific review?

What information needs to be included in the review?

Is the level of information provided enough and appropriate to allow for transparency so readers can judge the quality of the review?

Are the results clearly presented and explained?

Note(s): *Adopted from Snyder's (2019, p. 336) model “Important questions to consider in each step of the review'”

Table A1

The assessment of the MA dissertations' literature review quality*

MA dissertations No		1	2	3	4	5	6	7	8	9	10	11	12	13	14	15
Phase 1: Design
1	In relationship to the overall research field, is this literature review needed and does it make a substantial, practical or theoretical contribution?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
2	Are the motivation, the purpose, and the research question(s) clearly stated and motivated?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
3	Does the review account for the previous literature review and other relevant literature?	√	×	×	×	×	×	×	×	√	√	×	×	×	√	√
4	Is the approach/methodology for the literature review clearly stated?	√	×	×	×	×	×	×	×	×	√	×	×	×	×	√
5	Is this the most appropriate approach to address the research problem?	√	×	×	×	×	×	×	×	√	√	×	×	×	√	√
6	Are the methodology and the search strategy clearly and transparently described and motivated (including search terms, databases used, and explicit inclusion and exclusion criteria)?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
Total		6	2	3	3	2	3	3	3	5	6	3	3	3	5	6
Phase 2: Conduct
1	Is the search process appropriate for this type of review?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
2	Is the practical search process accurately described and accounted for?	√	√	√	√	×	×	×	√	√	√	×	×	√	√	√
3	Is the process of the inclusion and exclusion of articles transparent?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
4	Have proper measures been taken to ensure research quality?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
5	Can it be trusted that the final sample is appropriate and in concordance with the overall purpose of the review?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
Total		5	5	5	5	4	4	4	5	5	5	4	4	5	5	5
Phase 3: Data Abstraction and Analysis
1	Is the data abstracted from the article appropriate in concordance with the overall purpose of the review?	√	√	√	√	×	×	×	√	√	√	×	×	√	√	√
2	Is the process for abstracting data accurately described?	√	√	√	√	√	×	×	√	√	√	×	√	√	√	√
3	Have proper measures been taken to ensure quality data abstraction?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
4	Is the chosen data analysis technique appropriate concerning the overall research question and the data abstracted?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
5	Is the analysis process properly described and transparent?	√	√	√	×	×	√	×	×	√	√	×	×	×	√	√
Total		5	5	5	4	3	3	2	4	5	5	2	3	4	5	5
Phase 4: Structuring and Writing the Review
1	Is the review article organized coherently in relation to the overall approach and research question?	×	×	×	×	×	×	×	×	×	×	×	×	×	×	×
2	Is the overall method of conducting the literature review sufficiently described? Can the study be replicated?	√	×	×	×	√	×	×	×	√	×	×	×	×	√	√
3	Is the result of the review reported in an appropriate and clear way?	√	√	×	×	√	×	×	×	√	√	×	×	×	×	√
4	Does the article synthesize the findings of the literature review into a clear and valuable contribution to the topic?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
5	Are questions or directions for further research included? Are the results from the review useable?	√	√	√	√	√	√	√	√	√	√	√	√	√	√	√
Total		4	3	2	2	4	2	2	2	4	3	2	2	2	3	4

Note(s): *Adopted from Snyder's (2019, p. 338) model “Guidelines to assess the quality of a literature review”