Effectiveness of educational leadership through directors ’ performance in online higher education. The biggest online university in Spain

Purpose – Thestudyaimstoanalyzepedagogicale-leadershipinonlinehighereducationinSpainthroughthe application of VAL-ED at the International University of La Rioja. Design/methodology/approach – The methodology used to achieve the objectives has been a descriptive and quantitative methodology through a cross-sectional study based on the implementation of the questionnaire: Adaptation of the VAL-ED to the university context, developed by Palomino et al. (2022a). Findings – Indeed, the data obtained allow the authors to affirm how both directors and supervisors, as well as teachers, have been able to evaluate the leadership behavior of directors, compared to the competence standards of VAL-ED, having obtained very positive results that show how their leadership is, without a doubt, oriented to students. Given the fact that in the second specific objective the study sets out to determine to what extent the three groups of respondents (faculty, supervisors and directors) coincide in the effective performance of the pedagogical leadership of the directors, it can be highlighted that no great differences have been found in the responses of effectiveness obtained from directors, supervisors and teachers, since the results of the resulting scores among these three groups of respondents were reasonably similar.


Introduction
The use of information and communication technologies (ICT) nowadays is standardized in teaching-learning processes.This reality implies that educational organizations must be transformed to become flexible and innovative structures, open to new teaching methods and didactic resources, putting at the center the individual capable of reflecting and analyzing their environment, highlighting the constructivist methodology in which the student builds his own knowledge through autonomous work processes and the use of different technological tools (Palomino and Ramos, 2021).
In this context, virtual education is a reality characterized by being a teaching-learning process based on the principles of an active pedagogy, in which the student is responsible for participating frequently and effectively in different virtual environments (Aurangzeb and Mazhar, 2020).Unsurprisingly Chwen-Li et al. (2022) state that we can consider virtual learning as a new paradigm in the learning process, as it does not require face-to-face interaction and can be done without relying on a traditional classroom environment.However, as Cordie and Lin (2018) state, the prevalence of virtual learning leads to unique challenges, including dealing with various stakeholders, providing an effective product to students and establishing new paradigms on how to provide educational components.
Likewise, a new element stands out as an important part of this process: the reality of leadership.Indeed, leadership has emerged not only as a fundamental element for institutional success in achieving this transformation but also as a key factor in improving student performance.Not surprisingly, leadership in education is becoming a major factor in the improvement of education and a political priority.Indeed, in Marichal-Guevara et al. (2018), we find how various studies conclude that the exercise of leadership in education determines, in an important way, the quality of education.
When it comes to improving the quality and results of students in higher education institutions, different leadership models have been proposed; however, in of all of them, it is worth highlighting the role of pedagogical leadership in improving both the performance of students and the quality of universities in an effective way.

LODJ
That is why in higher education institutions that teach their studies in the online modality, we will refer to pedagogical e-leadership to refer to that model of pedagogical leadership that is developed in virtual environments, since this reality will also influence leadership in educational institutions, facilitating the student both to reflect and analyze their environment from a constructivist approach, in which he himself builds his knowledge through autonomous work processes (Ibarra et al., 2023).
Indeed, in virtual educational environments, the context of traditional leadership changes, since they allow us to develop and carry out a multitude of jobs, projects and tasks at any time and in any place, both in real and virtual spaces.In fact, some authors define e-leadership not only as an extension of traditional leadership but as a fundamental change in the way in which leaders and people relate to each other within organizations, as well as the way in which different organizations relate to each other (Tintor e and Gair ın, 2022).
We cannot ignore the fact that e-leadership takes shape in a context where collaboration and interaction, which seek to create and distribute an organizational vision in which goals, organizations and individuals are united, will be mediated by ICT (Antonopoulou et al., 2021).
Along the same lines, Avolio and Kahai (2003) state that, although e-leadership is a concept that emerged relatively early and with continued conceptual ambiguity, there will be significant differences between leading a traditional organization and leading those that have virtual environments mediated by ICTs, since these environments seem to require leaders who can deal with the paradoxes and dilemmas, as well as the complexity of the behaviors associated with the use of ICTs.It is essential to consider this nuance since educational e-leadership is based on the whole theory of leadership, but it does so assuming the characteristics of its nature and content.Miller and Ives (2020) state how e-learning has entered the mainstream of higher education as a strategic change agent, and this transformation requires e-learning leaders to develop the skills to successfully innovate at a time of increased competition and rapid technological change.
On the other hand, when we transfer the concept of leadership to the field of education, the concept of pedagogical leadership stands out, understood as that which is oriented toward improving student performance and results and is widely recognized as an important and effective element in improving educational quality.If, on the one hand, the first factor that has a significant impact on student learning is the teaching staff, the second element that can greatly influence the performance of a faculty, by facilitating a series of conditions and environments to improve the context in which it can better perform its work and students improve their learning, will be leadership (Anderson, 2010).
Not surprisingly, as Harris et al. (2022) assert, leadership in educational contexts can have a particularly positive influence on both institutional and student outcomes when it is distributed.Indeed, management and quality are related in many ways, and a strong leadership position enables distance education programs to achieve their goals successfully and efficiently while remaining responsive to the needs of their beneficiaries and users (Chang et al., 2022).
However, in a study conducted by Palomino et al. (2022a) in which they analyze the incidence of pedagogical e-leadership in higher education institutions from the most current scientific literature, we find how the scarce amount of work in this line indicates that research is still insufficient, thus evidencing the need to further deepen this area of research, which will revert in the efficiency and effectiveness of organizational development, as well as in the improvement of all dimensions of educational institutions of higher education.So, when it comes to understanding this reality, it is convenient to start from the fact that "Higher Education institutions are complex organizations in which their management is a challenge for leaders" (Smith, 2020, p. 39).

Pedagogical e-leadership in online education
That is why the need arises to continue the research work observing the necessary scientific standards that ensure the quality of knowledge, to understand to what extent the influence of learning management in higher education, and specifically in online training contexts such as The International University of La Rioja (from now on UNIR), is aimed at increasing and improving the performance of its students, as well as in improving the quality of institutions.This, then, is the research problem from where we start this study.
UNIR was created with the intention of allowing access to higher education to all people who, for different reasons, cannot study in person and who need to combine their learning with the different tasks they do every day.To achieve this goal, they have developed a unique methodology, with a strategic plan that seeks to ensure quality education.The features that define this methodology are the following: effective teaching focused individually on each student, adapting the pace of their training to their specific circumstances -UNIR provides a tutor to each student, to make it easy to provide individual tuition, finally, UNIR ensures the full commitment of faculty and management staff to the success of each student (UNIR, 2022).
Among the values that UNIR marks as a university, the following stand out: an offer of degrees and programs that meet the needs of society; careful attention to the quality of teaching and resources; commitment to student success; personalization of teaching tasks with the accompaniment of the student throughout their learning process; ability to respond to the expectations of stakeholders; and constant attention to innovation in teaching and learning methodology (UNIR, 2022).
UNIR has 7 faculties and schools, in which 154 degrees and official masters and 95 own degrees are taught.The Faculty of Education currently offers 15 degrees, 23 official master's degrees, 7 own master's degrees and 17 continuing education degrees.
The UNIR team is made up of teaching and research staff (PDI) and management and administration staff (PGA).Both work in the same direction so that everything works effectively in each sector of the university.

General
The study aims to analyze pedagogical e-leadership in online higher education in Spain through the application of VAL-ED at the International University of La Rioja.The Vanderbilt Leadership in Education Assessment (VAL-ED) is a multiple assessment tool, with an evidence-based rating scale, that assesses the behaviors of directors known to directly influence faculty performance and, through them, student learning (Vanderbilt University, 2011a, c).

Specific
(1) Identify the effectiveness of behavior in the leadership of the director taking as reference the competence standards of VAL-ED.
(2) Determine to what extent the three groups of respondents (faculty, supervisors and directors) agree on the effective performance of the pedagogical leadership of directors.
(3) Verify which areas of leadership behavior represent the greatest strengths, limitations and top needs for improvement

Methodology
The methodology used to achieve our objectives has been a descriptive and quantitative methodology through a cross-sectional study based on the implementation of the questionnaire: Adaptation of the VAL-ED to the university context, developed by Palomino et al. (2023).These authors highlight how, for decades, educational research has focused on finding ways to improve leadership effectiveness, valuing managerial evaluations as an important tool that can detect those improvements.However, developing these assessment tools has always been challenging, because they vary greatly from each other in both content and focus, as well as different perspectives on what leadership behaviors should be assessed.In response to this need, in 2005, a team of researchers from Vanderbilt University and the University of Pennsylvania, both in the United States, developed a valid, reliable and theoretically grounded assessment model of managerial educational leadership that has recently been adapted to the context of higher education.Notably, the VAL-ED assumes that effective student-centered leadership lies at the intersection of two dimensions: the core components built through key processes.
As we find in Palomino et a. (2022b), it is important to note that the VAL-ED has been designed to evaluate critical leadership behaviors to perform diagnostic analysis, performance feedback, progress tracking, professional development planning and summative evaluation.These authors highlight how there have been different studies in which the validity and reliability of the VAL-ED have been tested:  2014) also wanted to check the validity of VAL-ED in primary and secondary schools in the United States.They carried out a study in which they analyzed the accuracy with which the VAL-ED scores can identify the belonging to the two groups previously selected by the superintendents (directors whose performance of their functions was in the top 20% and the bottom 20%).Goldring et al. (2015) investigated the psychometric characteristics of the Vanderbilt Assessment of Leadership in Education, analyzing its convergent and divergent validity.They hypothesized that VAL-ED is highly correlated with other measures of instructional leadership.Likewise, in their study, Goff et al. (2015) highlight how numerous researchers have conducted several studies to validate VAL-ED, proving to be a reliable instrument that can be used in multiple contexts.Minor et al. (2017) evaluated the test-retest validity of the VAL-ED.They found that the principal and teacher grades of moment 1 and moment 2 have significant, positive and meaningful correlations.With these results, the Vanderbilt Evaluation of Leadership in Education (VAL-ED) instrument is, therefore, a reliable and valid instrument to measure the effectiveness of educational leadership.
On the other hand, we know that when assessment instruments are used as intended with a sufficiently significant sample for each of the groups that have interacted with a director, these instruments (in our case the adaptation of the VAL-ED in the context of higher education) will produce reliable and valid results on key leadership behaviors focused on learning, facilitating interpretations that will allow directors to discover areas for improvement that affect leadership focused on more effective learning (Palomino et al., 2022b).

Data collection tool
The questionnaire used to evaluate the pedagogical leadership of directors has been the translation and adaptation of the VAL-ED to the Spanish university context developed by Palomino et al. (2023).These authors considered theoretical and empirical antecedents, as well Pedagogical e-leadership in online education as the judgment of several experts in the subject of the universities of Granada and the International University of La Rioja to validate the adequacy of the questions in the university context.It was also validated by a graduate in translation and interpreting, a graduate in English philology and a graduate in Hispanic philology to guarantee the correct translation.
Since leadership focused on effective learning is at the intersection of two dimensions, the main components created through key processes, the VAL-ED has been designed to evaluate both the main components and the key processes.The main components refer to the characteristics of educational institutions that support student learning and improve the ability of the teaching staff to teach their students.On the other hand, key processes refer to how leaders create and manage those major components.

Main components
As we have already highlighted and as we find in Porter et al. (2008), the VAL-ED framework includes six main components that represent the constructs of learning-centered leadership: (1) High standards for student learning: These are defined as the degree to which leadership ensures that individuals, teams and academic goals are aligned to achieve rigorous learning academically and socially.
(2) Rigorous curriculum: It is understood as the content of the instruction and which must be ambitious.
(3) Quality instruction: As these authors point out, a rigorous curriculum by itself is insufficient to ensure optimal student learning, since quality instruction is also required, that is, an effective instructional practice that maximizes the academic performance and social learning of students.
(4) Culture of learning and professional behavior: Another major component of the assessment framework is leadership that ensures that the institution is organized, rather than from a bureaucratic point of view, as a learning community, in which student development from both an academic and social point of view is at the center.
(5) Connections to external community: Leading an institution with high expectations and academic achievement on the part of all students also requires strong connections with the community.
(6) Performance accountability: There is an individual and collective responsibility among the leader, faculty, students and community to achieve rigorous academic and social learning goals.

Key processes
Similarly, the conceptual framework presents six key processes which, although they are interconnected and are recursive and reactive between them, for the purposes of descriptive evaluation and analysis, are reviewed each individually.
(1) Planning is understood as the articulation of a shared direction, as well as the implementation of coherent policies, practices, and procedures, which helps to focus resources, tasks, and people.
(2) Implementation consists of putting into practice the necessary activities to achieve high standards of performance by students.

LODJ
(3) Supporting, In that leaders must create enabling condition and; ensure and use the financial, political, technological and human resources necessary to promote academic and social learning.Supporting is a key process that secures the resources needed to make core components available and well utilized.
(4) Advocating is understood as addressing the diverse needs of students, ensuring that policies in the educational institution do not pose or create barriers for certain students, as well as that students with special educational needs receive content-rich instruction.
(5) Communicating consists of the development, use and maintenance of information exchange systems between members of educational institutions and with their external communities.This communicating should inform, promote and link institutions being key in supporting students' academic and social learning.
(6) Monitoring involves the collection and analysis of data in a systematic way to make judgments that guide decisions and actions for continuous improvement (see Table 1).
According to empirical research (Hallinger and Heck, 1996;Leithwood et al., 2004), this assessment model does not foresee the direct effects of leadership behaviors on student success, but rather those leadership behaviors that will lead to changes in teacher performance that, in turn, will lead to student success (Murphy and Meyers, 2008).
Finally, the fact of being a 360-degree evaluation tool implies that the different key people around the director (that is, faculty, the director himself and the supervisors of the latter) will be the ones who answer the questionnaire to evaluate leadership.In order to verify the reliability of the questionnaires, Cronbach's alpha value was calculated for each of them, obtaining the following values: 0.896 for the Supervisors questionnaire, 0.948 for the directors/coordinators questionnaire and 0.938 for the teachers' questionnaire.Since all the values obtained are very close to 1, we can conclude that the three questionnaires obtained are reliable.

Sample
The sample has been obtained from the Faculty of Education of the International University of La Rioja.Of the 38 official degrees offered at the Faculty of Education of the International University of La Rioja, 11 participated in the study, obtaining the following sample of participants: 11 directors, being 3 degree directors and 8 directors of official master's degrees; 5 supervisors of the directors of the 11 participating degrees, with 2 supervisors of directors of bachelor's degrees and official master's degrees, and 3 supervisors of directors of official master's degrees; finally, a total of 89 professors from the 11 degrees participated in the study.
In this way, a participant sample formed by 105 people, representing 100% of the directors of the degrees; 100% of the supervisors of these, and more than 75% of the teachers who teach in each degree was obtained.We should take into account that neither the directors of the master's dissertation subject, nor the teachers of the practices subject, have participated.The sample represents in a heterogeneous way the different degrees of the Faculty of Education and the interested parts.

Procedure
To evaluate the leadership evidence of the three agents involved (faculty, directors/coordinators and supervisors), data collection was carried out between May 2021 and February 2022.In all, 105 questionnaires were collected.The instrument was applied in digital format, and the participants in the research were previously informed about the objective of the study.Their participation was voluntary, and responses confidential and anonymous.
To achieve the potential of VAL-ED for its intended purposes and to be managed as designed, as suggested by Vanderbilt University (2011a, b), the following aspects were considered in both the key usage summary and the implementation guide: (1) The teachers invited to complete the VAL-ED had to result in a representative and reasonably large sample.In this sense, we obtained 89 responses that, although they do not represent the 100% percentage of faculty members who should have responded to the surveys, we can consider as a sufficient number of responses to carry out the study.On the other hand, it should be noted that, since both the appropriate supervisors were included, together with the directors and the faculty members of the degrees, the result is a 360-degree evaluation.
(2) Similarly, the evaluation was coordinated by a neutral and objective person who did not complete the VAL-ED.
(3) It is recommended, on the other hand, that the evaluation should only be carried out, at the earliest, during the end of the second month of the academic year, as this increases the likelihood that respondents have had a reasonable opportunity to interact with the director they are evaluating.In this sense, it should be noted that the questionnaires were collected at the end of the second semester of the 2020-21 academic year and at the end of the first semester of the 2021-22 academic year.
(4) Teachers were guaranteed that their answers were anonymous, and they were allowed adequate time to read, reflect on the evidence, and rate the director's behavior.

Analysis and interpretation of results
As we found in Vanderbilt University (2011a, c), the VAL-ED behavior inventory provides information on a total score, six subscales for core components, six subscales for key LODJ processes separately for each group of respondents and an overall average among the groups of respondents.
The most important score that results from the VAL-ED is the overall total effectiveness score of the director.This score is based on the average ratings of all respondents, where each group of respondents has the same weighting and is reported on the 5-point effectiveness metric used to rate each of the 72 elements of the instrument.Therefore, the director's overall total effectiveness score and core component and key process subscale scores are reported on a continuous scale from a minimum of 1.0 (non-effective) to a maximum of 5.0 (very effective).The various effectiveness subscale ratings for core component and key process behaviors fall within this same operating range.

Comparison of proficiency standards with cut-off scores
Vanderbilt University (2011a, c) highlights that to have an interpretative framework that allows us to identify the level of competence of the director, it is necessary that we define each of them.For this, a series of cut-off scores will be established, which will establish each of the levels.
(1) Lower than basic: A leader at the sub-core level displays leadership behaviors in key components and processes at levels of effectiveness that, over time, are unlikely to influence faculty and student performances.
(2) Basic: A leader at the basic level of competency displays leadership behaviors in key components and processes at levels of effectiveness that, over time, are likely to influence and result in acceptable added value for performance for some, but not all, groups of students.
(3) Competent: A competent leader exhibits leadership behaviors in key components and processes at levels of effectiveness that, over time, are likely to influence the faculty and result in acceptable added value for the performance of all students.
(4) Distinguished: A distinguished leader exhibits leadership behaviors in key components and processes at levels of effectiveness that, over time, are sure to influence the faculty and result in strong added value for the performance of all students.
The three cut-off scores used to differentiate the four levels of leadership proficiency are as follows: 3.29 will determine the boundary between the basic and lower levels than basic; 3.60 between basic and competent; and 4.00 between competent and distinguished.The result of these cut-off scores implies that directors who score in the range of 1.0-3.28 will be rated with leadership behavior at the "lower than basic" level.Directors who score average in the range of 3.29-3.59will be described as leadership behavior at the "basic" level.Directors who score an average response in the range of 3.60-3.99will be described as leadership behavior at the "competent" level.Finally, directors who score average in the range of 4.00-5.00will be described as leadership behavior at the "distinguished" level.

Results
As discussed before, the VAL-ED behavior inventory provides information on a total score, six subscales for core components, six subscales for key processes separately for each group of respondents and an overall average among the groups of respondents.The core components and key processes are based on the same information, so while their information is redundant, the two separate profiles offer diagnostic information on how a director's Pedagogical e-leadership in online education behavior can be improved to achieve a more effective institution and, in turn, improve student performance.
The data obtained allow us to analyze the pedagogical e-leadership in the Faculty of Education of the UNIR, as well as identify the effectiveness of behavior in the leadership of the directors taking as reference the competence standards of VAL-ED, as well as verify which areas of leadership behavior represent the greatest strengths, limitations and main needs for improvement.
As can be seen from Tables 2 and 3, scores obtained at the intersection of the "high standards for student learning" core component with each of the key processes result in distinguished performance by directors in each of them.The scores obtained at the intersection of the "rigorous curriculum" core component with each of the key processes result in a distinguished performance in all these, except for the key component "planning", in which the performance would be competent.
Regarding the scores obtained at the intersection of the main component "quality instruction" with each of the key processes, it is worth noting how we find a distinguished performance in all of these, apart from the key component "Monitoring", in which the performance would be competent.In addition, scores obtained at the intersection of the main component "culture of learning and professional behavior" with each of the key processes result in distinguished performance by directors in all of them.
It is noteworthy, however, how the scores obtained at the intersection of the main component "Connections to external community" with each of the key processes result in competent performance in each of them, except for the key component "Supporting", in which the performance would be basic.Finally, the scores obtained at the intersection of the main component "performance accountability" with each of the key processes again result in distinguished performance by directors in all of these.
When we compare the performance results obtained by each of the groups of respondents directors, supervisors and faculty (Tables 4-6)to determine to what extent the three groups of respondents (faculty, supervisors and directors) agree on the effective performance of the pedagogical e-leadership of the directors, as we proposed in the second specific objective, we observe how the results of the interaction of the main components "standards for learning of high students" and "rigorous curriculum", coincide with all of them obtaining a distinguished assessment in the performance of the directors.
In the case of the main component "quality instruction", the level of performance obtained in four of the key processes coincides, finding differences in the key processes of planning and monitoring in the case of the group of respondents of teachers; and advocating and monitoring in the group of respondents of supervisors, who rate them with a performance as competent, compared to the distinguished with which directors are evaluated in all of them.
When we analyze the results of the main component "culture of learning and professional behavior," the level of performance obtained in the six key processes coincides when we compare the responses of the groups of directors and teachers.Both groups qualified it with the level of distinction.However, when comparing the scores of the group of directors with that of supervisors, we find differences in the key process of monitoring: while directors rate performance with the level of distinguished, the group of supervisors qualifies it as competent.
It is in the results of the main component "connections to external community" that we find greater differences in the performance scores of each of the key processes.The first thing that stands out is that it is the component in which the lowest scores have been obtained in all groups of respondents.On the other hand, it highlights how it is the supervisors of the directors and the directors group of respondents that give the lowest scores to the performance of the key processes of this component, since the highest level of performance obtained is that of competent in the communicating process, having being valued with the lower than basic       6. Results scores VAL-ED 360 of the teachers in the intersection matrix of each main component and the key processes LODJ level.The teachers value a competent performance in four of the processes evaluated: implementing, supporting, communicating and monitoring, compared to the three that we find in the group of directors: implementing, supporting and advocating.
It should be noted that in the main component "responsibility for performance," it is the group of teachers that give the lowest score to the planning process, qualifying it with a competent performance, compared to distinguished that the groups of directors and supervisors grant.In the rest of the processes of this component, all groups of respondents agree in their evaluation, evaluating the performance of directors with a distinguished level.
As we have discussed before, while it is true that the core components and key processes are based on the same information, studying the two separate profiles offers diagnostic information on how a director's behavior can be improved to achieve a more effective institution and, in turn, improve student performance.From the results obtained (Table 7), it can be seen that the level of performance of directors in the main components is distinguished, except for the component "connections to external community" that obtains a level of performance of competent.
On the other hand, when analyzing the performance level of the key processes (Table 8), we observe how the level of competence obtained is distinguished in all of them.
Polikoff et al. (2009) evaluated the differential operation of VAL-ED elements.Porter et al. (2011) studied the validity and reliability of VAL-ED through a nationwide study in the United States.Minor et al. (