Structural variable validation of an Online Learning Response Behavior (OLRB) instrument: A comparison analysis of three extraction methods of Exploratory Factor Analysis

Purpose – Measuring internal response of online learning is seen as fundamental to absorptive capacity which stimulates knowledge assimilation. However, the evaluation of practice and research of validated instruments that could effectively measure online learning response behavior is limited. Thus, in this study, a new instrument was designedbasedonliteraturetodeterminethestructuralvariablesthatexistintheonlinelearningresponsebehavior. Design/methodology/approach – A structured survey was designed and distributed to 410 Malaysian students enrolled in higher-education institutions. The questionnaire has 38 items, all of which were scored using a seven-point likert scale. To begin with, exploratory factor analysis with three types of extraction methods (i.e. principal component, principal axis factoring and maximum likelihood) was used as the method for comparing the outcomes of each extraction method ’ s grouping variables by constantly using a varimax rotationmethod. Inthe second phase,reliability analysis was performed to determinethe reliabilitylevel of the groupingvariables, and finally,correlation analysis was performed to determine the discriminant nomological validity of the grouping variables. Findings – The findings revealed that nine grouping variables were retrieved, with all items having a good value of factor loading and communalities, as well as an adequate degree of reliability. These extracted variables have good discriminant and nomological validity, as evidenced by correlation analysis, which confirmed that the directions of relationships among the extracted dimensions follow the expected theory (i.e. positive direction) and the correlation coefficient is less than 0.70. Research limitations/implications – This study proposes a comprehensive set of questionnaires that measure the student ’ s online learning response behavior. These questionnaires have been developed on the basis of an extensiveliterature review and have undergone a rigorousprocess ofvalidity and reliability Originality/value – This study ’ s findings will aid academic practitioners in assessing the online learning response behavior of students, as well as enhancing the questionnaire ’ s boost factor when administered in an online learning environment.


Introduction
In a typical setting, universities and colleges are urged to incorporate blended learning for all academic programs. Blended learning involves a blend of traditional face-to-face training with digital learning. In blended learning classes, the learning materials were primarily provided through face-to-face interaction, but they were also accessible through a robust learning management system (LMS) to support and increase instructor-student and student-student connection after class. However, as of March 2020, all higher-education institutions in Malaysia have been shuttered owing to the COVID-19 epidemic. During the implementation of this measure, students will still be in the beginning of the semester and will be required to transition to an online learning mode. This has resulted in a rapid shift to online instructional delivery. To assist in limiting the spread of the virus that causes COVID-19, some schools have elected to cancel all face-to-face sessions, including laboratories and other learning opportunities, and have demanded that teachers transfer their courses online. Globally, the majority of colleges and institutions outside of Malaysia have used the same approach as one of the restraints to minimize the COVID-19 viral transmission process.
Due to this, digitalization has contributed to a surge in interest among educational scholars worldwide, particularly in developing nations and the Asian region. Responses to technology adoption are characterized by extremely disparate attitudes, digital skills and degrees of readiness (Yasin et al., 2020). The phenomenon has also led to an emphasis on behavioral intention in the context of individual components such as cognition and motivation (Costley and Lange, 2017;Oluwajana et al., 2019), in addition to learning tools (Wammes et al., 2019). In the context of this study, behavioral engagement is demonstrated by students' participation such as idea generation, participation in the classroom and extracurricular activities, concentration, assignment submission and adherence to the instructor's directions (Reschly and Christenson, 2012). All these variables are associated with a higher number of behavioral interventions and educational tools for students to increase their involvement in online classrooms for successful learning (Hughes et al., 2020). Consequently, it is vital to research how novice digital learners react to learning.
Although there are similarities in the research issues on behavioral response, the approaches used to investigate students' perspectives on learning behavior have not been consistently verified. Therefore, educational researchers utilizing digital platforms must have access to instruments or measures that have been cross-validated across varied groups (Guo and Liu, 2018;Hsiao et al., 2019). This would increase the reliability, generalizability and practicality of online learning studies.
Still, validated instruments are used rather infrequently in research on the response behavior of students to online learning (Yasin et al., 2020). As a result, the purpose of this study is to investigate the procedures that may be used to determine the validity and reliability of the instrument. Evaluation of students' abilities to comprehend, produce and evaluate information offered within the context of a Malaysian online educational setting is carried out. The same method is advantageous for designing evaluation tools to measure other aspects of a student's readiness, such as their perspectives and preferred learning style. Consequently, we established two aims for this study: (1) to identify significant factors for assessing the online learning response behavior in higher-education institutions and (2) to assess the structural variable validity and reliability of the online learning response behavior questionnaire.

Study design
In this study, both a cross-sectional applied survey and a combination of quantitative methodologies were utilized. According to Creswell (2014), the purpose of this study is to validate and investigate the structural aspects of the targeted structures, and hence, these two research methodologies were chosen. Moreover, the phrase of the operational items in the Likert scale that were adopted from the previous study was amended in accordance with the study context. Therefore, it is vital for the researchers to revalidate this instrument using a suitable quantitative technique (Hair et al., 2010).
This study is particularly interested in Malaysian public university students having at least one year of experience using an online learning platform during the Movement Control Order (MCO). In the lack of a sample frame for selecting respondents, this study used a combination of convenience and judgmental sampling methods, both of which are regarded appropriate (Creswell, 2014).
There were 410 Malaysian public university students who volunteered to participate in this study by responding to a social media-distributed questionnaire. The sample size for this study is sufficient since the minimal sample size for a large population (1m or more) is 384 (Krejcie and Morgan, 1970). In addition, Hair et al. (2010) and Tabachnick and Fidell (2007) recommend evaluating at least 350 exploratory factor analysis (EFA) samples. Consequently, a sample size of 410 respondents may be considered adequate for this study.
2.2 Process of selecting the sources of a questionnaire using systematic review The process of constructing a questionnaire begins with the collection of accessible definitions, concepts and practicalities of the key notion of students' learning response behavior from the previous academic literature. This procedure begins with a search of the most cited sources pertaining to the assessment of learning response behavior in the Scopus database, which contains only peer-reviewed scientific publications, in an effort to exclude duplicate sources.
Using the Boolean search operator "TITLE ABS-KEY" ("learning response behavior"), a total of 167 relevant sites were retrieved (survey OR scale OR questionnaire OR model). From these, 108 relevant sources remained because they met the selection criteria for the inclusion of the studies, which are as follows: (1) The research is published in a scientific publication.
(2) The study consists of empirical research with survey-based learning response behavior measures.
These 108 relevant sources were retrieved via the digital library service provided by the Universiti Teknologi MARA or via the open access option, with the material regarding survey-based assessment of learning response behavior being the only topic of interest. From a total of 108 suitable sources, 101 were eliminated from this procedure for the following reasons: (1) Title and abstract irrelevance (15 papers); (2) Conceptual papers (23 papers); (3) Papers that employ a qualitative technique (43 papers); Online learning response behavior instrument (4) The articles' content did not address the optimal usage of questionnaire methodology (13 papers); (5) The papers were not authored in English (7 papers).
Following the screening procedure, seven papers that met the aforementioned selection criteria were included in the systematic review. Figure 1 depicts the flow diagram of the preferred reporting items for systematic reviews and meta-analyses (PRISMA)-based search procedure used in this research. We integrate two major types of methodological criteria in the process of framework or item assessment: evaluation of a survey-based operationalization measurement scale ( Table 1) and evaluation of measurement quality ( Table 2). These stages, according to Carmines and Zeller (1979), are required but not sufficient for obtaining a trustworthy and valid structural variable instrument. Based on these two methodological criteria, all seven of the selected studies fit the requirements, with 100% of the criteria met (Table 3).

Questionnaire items
To evaluate the learning response behavior of students, the researcher developed instruments based on a systematic review analysis that matched the PRISMA criteria and two major methodological criteria, as shown in Table 3. A 38-item survey was modified using a seven-point Likert scale, where 1 corresponds to strongly disagree and 7 corresponds to strongly agree, and was randomly distributed in the instrument setup. Adapted from Yasin et al. (2020), three elements of the student's learning readiness were used (i.e. Q1, Q5, Q6). Based on Gao et al. (2020), the student's learning engagement is defined as the reflection of cognitive and emotional feelings representing their behavior in response to an online learning environment consisting of seven modified items with two dimensions: cognitive engagement (i.e. Q2, Q7, Q9) and emotional engagement (i.e. Q2, Q7, Q9) (i.e. Q3, Q4, Q8, Q10). This instrument consists of eight items to assess the learning motivation construct referred to as the beliefs, intentions and feelings that motivate students to learn, modified from the study by Liu (2020): four items for intrinsic motivation (i.e. Q12, Q13, Q16 and Q19) and four items for extrinsic motivation (i.e. Q11, Q14, Q15 and Q17). Based on the tles and abstracts, 15 publica ons were omi ed due to unrelated themes.
86 full-text ar cles were excluded because conceptual papers, using qualita ve approach, lack of mythological aspects, and not wri en in English Figure 1.
Flow diagram of the searching procedure using PRISMA procedure AAOUJ Alternative assessment, on the other hand, is the variable used to evaluate students' learning and support their cognitive development by regularly examining and providing feedback on their work. Thus, eight questions (i.e. Q21, Q24, Q25, Q27, Q30, Q31, Q33 and Q35) were adopted from the study by Leeuwenkamp et al. (2019). This study also modified the Awidi and Paynter (2019) student's learning experience, which is defined as the process of knowledge construction and converging ideas developed through social interactions, with four items (i.e. Q22, Q23, Q29 and Q32), whereas the remaining four indicators were adapted from the study by Utriainen et al. (2018) to evaluate the critical thinking learning experience (i.e. Q20, Q26, Q28 and Q37). This instrument also incorporated students' response behavior toward online learning, adapted from the research of Bui et al. (2020) ( 1 ) Criteria code refers to Table 1 and Table 3. (1)   Online learning response behavior instrument Q36, Q38 and Q18). The questionnaire is supplied in Appendix. Importantly, the premise of utilizing this questionnaire as an elicitation tool is that the unit of analysis must have online learning experience as the purpose of this questionnaire is to measure the respondent's online learning response behavior. In addition, the responder must be an adult learner in order to provide an accurate response to each questionnaire item as the items included in this questionnaire tend to be psychological measures that require exact evaluation.

Statistical analysis
The IBM SPSS version 21.0 statistical software was used for data input and analysis methods. The EFA and correlation analysis were used because the primary purpose of this study was to validate and investigate the structural variable validity and reliability of the instrument after undergoing the process of sentence modification and Likert-scale use by the original authors (Hair et al., 2010;Tabachnick and Fidell, 2007). As recommended by Hair et al. (2010) and William et al. (2010), this form of study enables the researcher to validate and modify the employed indicators (2010). In this work, three EFA extraction methods were utilized to compare the instrument's valid structure. The first extraction method that was used is a combination of principal component (PC) extraction method with a varimax rotation (PC þ varimax), the second and third extraction methods were principal axis factoring (PAF) extraction method with a varimax rotation (PAF þ varimax) and the final extraction method was maximum likelihood extraction method with a varimax rotation (ML þ varimax).
Using a Kaiser-Meyer-Olkin (KMO) index, the existence of a nonidentical matrix and the sufficiency of the covariance matrix among the indicators must be determined at the initial phase of the study. As proposed by Field (2009) and Pallant (2010), this index value must exceed 0.60. As a support for this conclusion, Bartlett's test of sphericity must be significant (Pallant, 2010); hence, it confirms the existence of a nonidentical matrix and that the covariance matrix among the indicators is adequate for the EFA. Thompson and Daniel (1996) suggested that, in the second stage of this study, a researcher should use several criterion techniques to identify the number of dimensions that should be retrieved from the constructs and to check and validate the number of dimensions that exist in the constructs. Therefore, only dimensions with eigenvalues more than 1.00 are retrieved from the analysis, and the total percentage of variance extracted should be greater than 60%, as recommended by Hair et al. (2010), Field (2009), Tabachnick and Fidell (2007 and Thompson and Daniels (1996). In addition, Watkins (2000) recommends using the Monte Carlo PCA parallel analysis simulation eigenvalues to calculate the number of extracted variables. In this study, only extracted variables having an eigenvalue larger than the eigenvalue of the Monte Carlo simulation are considered extracted.
Examining the value of factor loading and communalities from each extraction technique (such as PC þ varimax, PAF þ varimax and ML þ varimax) is the next step in validating the indicators under each dimension extract. According to Field (2009), for an exploratory study on refining and validating indicators using the EFA, the values of factor loading and communalities must be more than 0.40 and the sample size must exceed 350. This is a crucial step as it ensures that the retrieved dimensions are different, meaningful and valid with a sufficient degree of confidence. In the last step of the EFA methods, a Cronbach's alpha reliability test was used to evaluate the internal consistency of the indication group. Nunnally and Bernstein (1994) indicated that a threshold of 0.70 or above can be used to show the reliability of grouped elements.
As proposed by Creswell (2014), Hair et al. (2010) and Kaptein (2008), discriminant validity and nomological validity were assessed following the EFA and reliability analysis. Using correlation analysis, they describe nomological validity as the act of examining the logical and meaningful link between dimensions that follows the predicted direction of association (i.e. positive relationship or negative relationship). In addition, Hair et al. (2010) and Kaptein (2008) used the correlation coefficient to assess discriminant validity using correlation analysis. Observation suggests that when the correlation coefficient is less than 0.70, the extracted dimensions have discriminant validity.

Data analysis and results
3.1 Convergence validity from the EFA and reliability analysis The findings of multiple criterions for determining the number of variables to be retrieved from this instrument's 38 total items are summarized in Table 4. The study shown in Table 4 revealed that the EFA analysis for these three extraction techniques should extract nine variables from a total of 38 items since the first nine eigenvalues under Kaiser's criteria (e.g. range: 1.059 to 5.762) were greater than 1.00. Moreover, when comparing these Kaiser's eigenvalues with simulation eigenvalues from the Monte Carlo analysis, the same result occurs: only the first nine Kaiser's eigenvalues are greater than the first nine simulation eigenvalues from the Monte Carlo analysis, confirming that nine variables should be extracted from a total of 38 items. The cumulative percentage of variation explained for these nine extracted variables is 83.83%, indicating that the extraction of these nine variables from a total of 38 indicators is legitimate. Bartlett's test of sphericity for the items was likewise highly significant (X 2 5 20419.75, p < 0.01). On the other hand, the KMO index for these three extraction techniques of the EFA was 0.974. Therefore, the covariance matrix for these 38 items was not a matrix of identity. Consequently, all elements in the instruments can be used for the EFA if all three extraction techniques are included. Table 5 displays the factor loading and communality value for each item based on the three extraction methods used in the EFA. As with the first EFA extraction approach (i.e. PCA þ varimax), the analysis revealed that all items had loading (range: 0.504 to 0.903) and communality (range: 0.709 to 0.963) values greater than the minimal threshold value of 0.40. Alternatively, the second extraction technique of the EFA (i.e. PAF þ varimax) yields the same result, with all 38 items having loading (range: 0.439 to 0.874) and communality (range: 0.508 to 0.915) values greater than 0.40. The loading (range: 0.387 to 0.948) and communality (range: 0.373 to 0.919) values were just below 0.40 for the third extraction technique of the EFA, ML þ varimax. Since the sample size for this research (n 5 410) is regarded to be substantial, the ML þ varimax technique's findings can be considered reliable. Therefore, it is possible to infer that all 38 items were valid and could be utilized to measure the intended components in this study.  Based on the grouping pattern of indicators, these three EFA extraction techniques exhibit a similar pattern. Student's learning readiness, student's perception of alternative assessment provided and student's response behavior toward online learning factors were discovered to have maintained their one-dimensional structure. Based on these three extraction analysis approaches, all analyses demonstrate that the dimensions of the student's learning engagement, student's learning motivation and student's learning experience variables, each of which has a two-dimensional structure, were preserved. Cronbach's alpha scores for the grouped indicators ranged from 0.902 to 0.965, suggesting that all nine grouped indicators meet the minimal threshold value of 0.70. Therefore, it can be stated that all grouped indicators were legitimate and measured the variables consistently.

Discriminant and nomological validity from the correlation analysis
The nomological and discriminant validities of the extracted variables were examined using correlation analysis, and the results are presented in Table 6. At the 5% level of significance, the analysis revealed a substantial positive association between the extracted variables. All extracted variables may thus be confirmed to have good nomological validity since all bivariate correlations correspond to the predicted bivariate connection, which is a positive relationship. In addition, correlation coefficients for the analysis ranged from 0.410 to 0.667% The negative signs in the loading values indicate that these items were located at the negative axis of the rotation method; all three extraction methods of the EFA produce the same value of eigenvalues and hence produce the same variance explained from the extracted components which is 83.83%; PC 5 principal component; PAF 5 principal axis factoring; ML 5 maximum likelihood; λ 5 loading value; δ 5 communality value; refer to Appendix for code item definition Table 5. (1) ( (1) 5 student's learning readiness; (2) 5 cognitive engagement; (3) 5 emotional engagement; (4) 5 intrinsic motivation; (5) 5 extrinsic motivation; (6) 5 student's perception toward alternative assessment given; (7) 5 competency experience; (8) 5 critical thinking experience; (9) 5 student's response behavior toward online learning; n 5 410; *p < 0.05 Table 6. Correlation analysis among extracted dimensions from the EFA Online learning response behavior instrument when examined using the correlation coefficient. Since all correlation coefficients are less than 0.70, it can be concluded that the extracted variables provide a satisfactory level of discrimination between the variables.

Conclusion
This study illustrated the statistical processes for verifying, investigating and comparing the variable structures in the survey instrument utilizing three extraction techniques of the EFA and correlation analysis for the online learning response behavior instrument. Based on these three extraction techniques of the EFA, the validity of all items utilized to measure the targeted variables is satisfactory. This is due to the fact that all three different extraction methods for the EFA (i.e. PC þ varimax, PAF þ varimax and ML þ varimax) show a similar pattern of item grouping after undergoing the process of item sentence modification and Likert-scale weighting at the start of instrument development. In addition, the examination of reliability revealed that all extracted variables had sufficient and satisfactory internal consistency, as all Cronbach's alpha values were greater than 0.70. The correlation analysis also confirmed that all extracted variables from the EFA have good nomological and discriminant validity as the bivariate relationship between the extracted variables was significant and followed the expected direction of the relationship, and all correlation coefficients were less than 0.70.
On the other hand, based on the grouped elements from the EFA, student's learning readiness may be viewed as a continuous process that shows the students' mental preparation to accept the diverse method of technology-based learning (Yasin et al., 2020). According to Gao et al. (2020), student's learning engagement may be characterized as a technique that guides students in cultivating the instructor-acquired information and can be classified as cognitive engagement and emotional engagement. The fourth and fifth grouped factors, internal and extrinsic motivation, respectively, can be considered the total learning motivation of a learner. In this study, extrinsic motivation is defined as motivating students to attain their objective by providing an external incentive throughout the learning process on an online platform in the hope that students will acquire a strong interest in learning via the online platform (Liu, 2020). In contrast to extrinsic motivation, the intrinsic motivation variable was measured by the internal needs of the students during the process of learning on an online platform, where internal incentives such as creating an enjoyable environment during the learning process are anticipated to bolster intrinsic motivation among the students (Liu, 2020).
On the basis of the quality of alternative assessments offered by instructors, students' perceptions of alternative assessments may be interpreted as their perceptions of their learning progress and information about their desired learning goals (Leeuwenkamp et al., 2019). Experience with competencies and critical thinking are two criteria that may be used to assess a student's learning experience. This variable represents the entire learning experience encountered by the learner during the learning process. According to Utriainen et al. (2018) and Awidi and Paynter (2019), a student's learning experience can also be viewed as the construction of knowledge by the student through interaction with others or as the development of competencies and the capacity to apply knowledge to other contexts, which is critical thinking experience. Finally, student's reaction behavior toward online learning variable may be described as the behavior of students continuing to use the online learning platform as the learning medium in the future and introducing the online learning platform to their social circle as the optimal means of learning.

Discussion
Based on the findings of the EFA and correlation analysis, this online learning response behavior instrument building procedure is suitable for measuring the intended constructs. AAOUJ Due to the present COVID-19 epidemic, this study method may also be categorized as significant processes as this instrument can serve as an alternative beneficial tool for monitoring the entire online learning response behavior of students. This research also stressed the usefulness of the EFA and correlation analysis for verifying the survey instruments after the items have undergone a process of alteration from the perspectives of convergent validity, discriminant validity and nomological validity.
In addition, the outcomes of this study suggested that this questionnaire has the potential to be utilized internationally, not just in the Malaysian higher-education institutions setting, because most higher-education institutions worldwide have shifted to an online learning platform. In addition, all the factors used to quantify the online learning response behavior occur in the majority of higher-education institutions. This questionnaire is simple to administer because it consists of 38 items that may be categorized as small. Consequently, the likelihood of administrative data inaccuracy is likewise minimal. Since the structural themes of all variables revealed in this study lean toward the positive theme, the interpretation of the variables should likewise be positive. Using the average score approach, the score of each variable's structure must be high for a positive interpretation.

Limitations
In reference to this research, one of the drawbacks of this study is the lack of a causal analysis to identify the global questionnaire fit indices. This study thus recommends a causal analysis to further explore the global fit indices of the instrument's validity by employing a covariance-based confirmatory factor analysis or partial least square consistency factor analysis. In addition, for the purpose of calculating an overall global fit index, this instrument may be administered to secondary students and private higher-education students in both urban and rural regions. Moreover, our systematic review approach is restricted to terms that assess just the learning response behavior, and only seven papers are best accessible for reviews. Therefore, it is advised that, for future study, a more comprehensive word, such as students' response behavior, and a greater number of publications should be chosen for examination. for my subject Q21 In general, at this moment I perceive that, testing and assessment in online platform have a positive effect on my learning Q22 As a learner I see myself as highly competent in using online platform technologies Q23 My learning experience in this online platform has made me feel confident as a learner Q24 In general, at this moment I perceive that, testing and assessment in online platform add value to the time I have spent on the work done Q25 In general, at this moment I perceive that, testing and assessment in online platform are valuable instances of learning in their own right Q26 My learning experience in this online platform has make me able to evaluate issues critically for my subject Q27 In general, at this moment I perceive that, testing and assessment in online platform me to continue learning Q28 My learning experience in this online platform has make me able to apply theoretical knowledge to practice Q29 My learning experience in this online platform has motivated me to learn more Q30 In general, at this moment I perceive that, testing and assessment in online platform help me to navigate my own learning process Q31 In general, at this moment I perceive that, testing and assessment in online platform are geared towards the retention of my competencies in the longer period Q32 My learning experience in this online platform has fully engaged me as a learner Q33 In general, at this moment I perceive that, testing and assessment in online platform prepare me well for future learning activities Q34 I will make use the online platform regularly in the forthcoming time Q35 In general, at this moment I perceive that, testing and assessment in online platform give me confidence to continue learning Q36 I intend to make use the functions of online platform for providing assistance to my academic activities Q37 My learning experience in this online platform has make me able to develop new ideas Q38 I will give out my recommendation to others to use the online platform