University scholarship activities on module evaluation questionnaires at a British University

Purpose – The purpose of this study comprises the following three: (1) to ascertain the purpose of university moduleevaluationquestionnaires(MEQs)anditsreliability;(2)toevaluateUniversityX ’ sMEQ;and(3)tooffer howUniversitiesmay beabletosupporttheirteachingstaff withscholarshipactivitiesusing the MEQproject. Design/methodology/approach – University MEQ purposes and its reliability were investigated using literature reviews. The University X ’ s MEQ seven statements were evaluated by three university academic staff. The study was conducted at a British university in South East of England. The duration of this interdisciplinaryprojectwasfortwomonthswhichwasauniversityinterdisciplinaryprojectbetween14/07/20and13/10/20. Findings – The purpose for MEQs includes (1) students ’ satisfaction; (2) accountability for university authority and (3) teaching feedback and academic promotions for teaching staff. The evaluation of University X ’ s MEQ indicated that MEQ questions were unclear which do not serve reliable student evaluation results. This topic may be of interest to University MEQ designers, lecturers, University Student Experience team, University Executive Board, University administrators and University HR senior management teams. Originality/value – The following three points are considered original to this study: (1) MEQ purposes are summarised by students, university authority and teaching staff; (2) the evaluation of a British University MEQ;(3)providessuggestionsonhowlecturers ’ scholarshipactivitiescanbesupportedbytheuniversity-wide initiative and umbrella network. These are practical knowledge for the faculty and administrators of higher education institutions which may be of use.


Issues under consideration
There has been a recent shift in the teaching staff's academic responsibilities related to promotion and academic employment in the UK Teaching staff's, especially lecturers', scholarship activities are more emphasised in addition to their teaching responsibilities. Scholarship activities are individual's professional development in teaching and learning, for example, journal publications, conference presentation, research-related activities and so on. Another shift of academic responsibilities can be observed in the recruitment in the lectures' job descriptions and their pay grade. Many teaching-focus positions were advertised in non-STEM subjects, which the teaching staff are not expected to do any scholarship activities. However, the current academic trend places more emphasis on scholarship activities even for non-STEM lecturers' positions. This sudden change affects individual academic staff. It affects their promotion and may bring anxiety and burden to the lecturers, who occupy the majority of the university academic community. But what is this shift pertinent to MEQs and how does it affect MEQs?
The next subsection describes the background of this University X's university-wide initiative and umbrella network.

Background of the MEQ project
The above shift in the teaching staff's academic responsibilities in the United Kingdom also affected at University X, when the emphasis started just before the Covid-19 lockdown in July Deputy Course Director-International Marketing MSc (Business School). On 23/07/20, the second MEQ meeting was held with the Deputy Course Director-International Marketing MSc and the Associate Dean of the Business School-International Marketing MSc (Business School). The Deputy Course Director-International Marketing MSc left the project after one month and the researcher completed MEQ project alone. On 06/08/20, the third MEQ meeting was held with the Associate Dean of the Business School. On 14/09/20, the researcher submitted the MEQ report to the University Survey Group on 14/09/20. On 13/10/20, the researcher was invited to present the MEQ Report and answered questions from the Survey Group. The University Surveys Group included the Pro Vice Chancellor for Education and Students, Associate Dean of the Business School and the Deputy Pro Vice Chancellor of Student Experience (i.e. University's Executive Board Group).

Research questions (RQs)
RQs comprised the following two RQ questions:

RQ1.
What is the purpose of University MEQs?

RQ2.
Are University X's MEQs reliable? RQ1 was investigated by literature reviews, and RQ2 was investigated by literature review and evaluation of MEQ seven statements.
Next section starts with the literature reviews of purpose and reliability of MEQs, which is followed by methodology, discussion and conclusion.
2. Literature review: purpose and reliability of MEQs 2.1 MEQs and SETs Student Evaluations of Teaching (SETs) may have two purposes: institutional and local.
Institutional SETs are what universities set the standard questionnaires and distribute them to all schools and departments, which is considered more traditional. The majority of SET research appears to discuss on the institutional level SETs. Institutional SETs are also often relied upon as evidence cases for promotion and are an important element of a teaching portfolio.
Local SETs, on the other hand, usually refer to what each school or department set its own questionnaires in addition to the Institutional SETs. Local SETs are implemented as a complement to the institutional SETs providing insight and feedback on specific disciplinary questions. A major shortcoming of the University MEQs suffers from low response rates and is inconsistent year on year or across the university.
University's SETs are also referred to as the Module Evaluation Questionnaires (MEQs). In the United Kingdom, MEQs are the summative surveys which run at the end of a term or level of study at University X. As it belongs to an institutional type, it is referred as "University MEQs". However, local practices have grown to address the need for student feedback and many schools and departments also run their own MEQs locally (we refer it as "local MEQs") in response to the limitations of MEQs.
MEQ stakeholders may include students, staff, politicians, alumni, funding bodies and the general public (Kearns, 1998). Among these stakeholders, students, staff and university authorities are the focus of our discussion.

Students
"Students as consumers or partners" (Bienefeld and Almqvist, 2004) and the concept of "consumer satisfaction" (Blackmore, 2009;Olivares, 2003;Titus, 2008) has brought into education. Students are considered to play an important part in evaluating tutors. Yet, some Module evaluation questionnaires students may not appreciate how the information is used and find MEQs a pointless exercise (Bassett et al., 2015).

University authorities
SETs provide universities evidence of internal quality-assurance processes and institutional accountability (Johnson, 2000). Institutional accountability consists of the myriads of either tangible or intangible expectations (Kearns, 1998), but accountability appears to be considered of paramount important purpose at universities not only due to the British government emphasis on university's accountability and transparency across high education sector (2017), but also accountability links to attractive concepts such as "quality", "autonomy" and "integrity", all of which give special attention and positive connotation (Stensaker and Harvey, 2011). The universities report stakeholders regularly and openly (transparency), by documenting (visibility) on the outcome of their effective teaching, research activities and institutional management.
University authorities may often have ulterior motives in the student feedback process. They might be thinking in terms of league tables, for example, the National Student Survey (NSS). Some may approach it as a Key Performance Indicator (KPI). This then might feed into management decision-making, such as hiring, firing, promotion, pay, rewards and so on: "The problem here is that a KPI may mask the real reason for the decision, which could be something subjective" (Salmon, 2009, p. 77).
As current emphasis on SET data is on quantitative rating (Likert-scale questions), which is the easiness to collect, analyse and present the SET data (Penny, 2003), this also contributes why the majority of institutions use SETs as evidence.

Staff
Staff refers to teaching staff, specifically lecturers. SETs have two major purposes for teaching staff. The main and obvious purpose of SET is feedback of teaching quality for individual tutors (Spooren et al., 2013). SETs give feedback not only to individual academic staff who engaged in teaching but also for supervisory School unit/course coordinators and Head of school.
Another important purpose is academic promotion (Becker and Watts, 1999) and professional career progression as a measure of quality monitoring, administrative policymaking (Penny and Coe, 2004). There is a link between SETs and tutors' promotion (Becker and Watts, 1999) and career progression, that is, input for appraisal exercises (e.g. tenure/promotion decisions). It implies a close link among SETs, academic promotion and HR department. Beran and Rokosh (2009) claim that 62% of the academic staff feel that Department Heads and Deans make proper use of SET reports. Historically, the majority of institutions have been using SETs for promotion since the 1970s when evaluations came into use for faculty personnel decisions in hiring purposes (Becker and Watts, 1999).
Two major biases related to the tutors' grading affecting reliability of SETs are validity hypothesis and grading-leniency hypothesis. Validity hypothesis is defined as "students who have learned more in the class will receive higher grades and will naturally rate the professor more highly because of the knowledge they have gained in the course" (Patrick, 2011, p. 241). However, Abrami et al. (1980) argue that the effect of grading standards on students' ratings is very small and inconsistent across different rating items.
Grading-leniency hypothesis is defined that "the professor's leniency in assigning grades favourably influences student evaluation scores" (Patrick, 2011, p. 241). Simply, tutors who give higher grades also receive better evaluation (Carrell and West, 2010;Johnson, 2003;Weinberg et al., 2009). Some researchers suggest that tutors "buy" good evaluations by giving high grades (Isely and Singh, 2005;Langbein, 2008;McPherson, 2006;McPherson and Todd Jewell, 2007). As career promotion is important for tutors, some tutors may strategically adjust their grades to please the students (Braga et al., 2014), and others may teach the test content as students might reward tutors (Braga et al., 2014).

Other variables which affect reliability of MEQs
Tutors who teach more advanced courses are rated more favourably than teaching lowerlevel courses (Moritsch and Suter, 1988;Cranton and Smith, 1986;Feldman, 1978;Marsh, 1980;Goldberg and Callahan, 1991). This is due to the nature of heterogeneity in class composition in lower-level courses. Heterogeneity makes it extremely difficult for tutors to tailor to the students' preferred teaching and learning experience to satisfy all. By attempting to suit the curriculum to the majority, the instructor may easily fall into the trap of mediocrity, which inevitably dissatisfies some students (Casey et al., 1995;Reid et al., 1981). Furthermore, Social Science and Humanities tutors are rated more favourably than STEM subjects (Cashin, 1992;Centra, 1993). Lastly, insufficient sample size (Marsh and Roche, 1997) also affects reliability.

Methodology
The below illustration summarises the relationship between the research questions, methods and design.

Evaluation of university MEQs
The following two points of the University MEQ seven statements were focus of evaluation: (1) whether purposes are clearly identified and (2) whether strengths and/or weaknesses of the statement were also examined in each statement. Any suggestions for changes or revisions were proposed if there are any. For the quality assurance purposes, this evaluation was carried out by three members of the MEQ project -The Deputy Course Director of International Marketing MSc (Business School), followed by the researcher and the Associate Dean of the Business School.

Literature review
The literature review involved two stages: the retrieval phase and the screening phase. In the retrieval phase, the keywords for literature review were "SETs", "reliability", "assessment" and "evaluation" using Google, Eric and ProQuest and PsycINFO. The researcher conducted all abstract and content analysis of individual article and collected initially 78 data list (60 journal papers, 17 books and 1 conference paper) from the field of psychology, education, business management and sociology (See Table A1 "Literature review list"). The dates of the journals ranged between 1978 and 2020. In the next phase, 55 qualified and chosen journals (42 journals, 12 books and 1 conference paper, which are not denoted by * in Table A1 list) were classified into the following five categories: student purpose; university authorities purpose; staff purpose; other purposes; and reliability. The selected articles and books may be biased as the researcher focussed inclusively on the specific keywords of this study for relevance. The purpose of literature review is to integrate all relevant literature related to MEQ purposes and reliability concerning students, staff and university authorities.

Results of MEQ seven-statement evaluation
The followings are the results of University MEQs having seven statements regarding its purpose, strength/weakness and suggestions:

MEQ statement 1: it was clear to me what I would learn and why
Purpose: To assess the clarity of the module content, but it is not clearly stated. This purpose may be suitable for all stakeholders' purposes.
Weakness: The combination of "what and why" makes students' responses invalid. Suggestion: Divide the question into the following three separate questions.
I -It was clear to me what I would learn from this module.
II -It was explained to me or was stated in the module handbook why I would need to learn the content of this module.
III -The learning journey was explained to me or was stated in the module handbook why I would need to learn the content of this module.
4.2 MEQ statement 2: the materials on canvas were useful and relevant Purpose: To assess the validity and relevance of the module content on the Virtual Learning Environment (VLE), but it is not clearly stated. This purpose may be suitable for all stakeholders' purposes. Weakness: One statement but includes two questions (useful and relevant). It is unclear to which content the students may answer to of whether students might find the content relevant but not useful and vice versa.
Suggestion: Suggest to replace this to one of the followings.
I -The module content on canvas was relevant to the teaching of this module.
II -I have found the module content on canvas useful.
III -The module content on canvas helped me to learn this module better.
4.3 MEQ statement 3: the recommended reading lists were appropriate and up to date Purpose: To assess the reading list quality, but it is not clearly stated. This purpose may be suitable for all stakeholders' purposes. HEED Weakness: One statement but includes two questions (appropriate and up to date). Reading lists may be too broad and vague.
Suggestion: Suggest to replace this to one of the following.
I -The reading list offered good sources that helped my learning with this module.
II -The reading list content was up to date.
III -The reading list offered a variety of resources to support my learning.
4.4 MEQ statement 4: I have received feedback on my work (including queries after teaching sessions, in person or by e-mail) which may have helped my understanding/clarified things I did not know, helped to explain a grade or identified areas for improvement Purpose: To assess feedback. This purpose may be suitable for all stakeholders' purposes, but it is not clearly stated. Weakness: Unclear. It asks students a mixture of multiple feedbacks in one statement. The statement refers to summative and formative feedback which can confuse students for their response. For example, a tutor may be very good with summative feedback on their module which supports students to benchmark their learning at the end of the instructional unit. However, they may fail to offer adequate formative feedback and monitor students' ongoing learning during the module.
Suggestion: Suggest replacing this to the following two-step approach questions. The Step I question is: I have received feedback for the assessments of this module. If answered yes to the Step I question, then one of the following three statements may be Step II question.
II-choice 1 -The feedback I received helped me to understand better, areas of improvement and my strengths.
II-choice 2 -The feedback I received explained why I was awarded with the mark on my work. II-choice 3 -I was provided with opportunities to discuss my questions with my tutor (written and/or verbal) during the semester.
4.5 MEQ statement 5: clear information about the assessment of this module, and the marking criteria, was provided Purpose: To assess the assessment instruction and marking criteria, but it is not clearly stated. This purpose may be suitable for all stakeholders' purposes. Weakness: One statement but includes two questions whether information on assessment of the module was provided and if marking criteria were provided.
Suggestion: To keep the question.
4.6 MEQ statement 6: teaching accommodation and facilities were satisfactory Purpose: To assess the learning environment, but it is not clearly stated. This purpose may be suitable for all stakeholders' purposes. Weakness: the term "satisfaction" and "teaching accommodation and facilities" may be misleading whether it refers to learning content or physical building accommodations. This affects reliability and validity.
Suggestion: It needs confirmation regarding the meaning of "teaching accommodation and facilities". 4.7 MEQ statement 7: overall I was satisfied with the module Purpose: To assess the students' satisfaction, but it is not clearly stated. This purpose may be suitable for all stakeholders' purposes. This affects reliability and validity.
Weakness: The term "satisfaction" may provoke different connotations and sentiment by different cultures and individuals.
Suggestion: Suggest replace this to one of the followings.
I -Overall, I feel I have learned from this module.
II -Overall, I approve the quality of teaching on this module.
III -Overall, I approve the quality of the content of this module.
IV -Overall, I approve the activities included in this module.

Purpose
The purpose of seven statements was unclear. As there are only seven statements, it is assumed that the University MEQs' designers tried to include two questions in each statement, which resulted in unclear and vague questions, resulting students' various interpretations of the questions and answers. For example, questions such as "how satisfied are you . . ." do not often lead to productive answers and fail to assess teaching quality. To ensure that the intended meanings are communicated, consideration should be given to students' perception of the words such as "satisfied" as it may give different meanings depending on the students and cultures. Careful consideration should be given to the design and focus of the questions.

Reliability
Reliability is defined as "stability or consistency with which we measure something" (Robson, 2002, p. 101) and "for a research instrument to be reliable, it must be consistent" (Gray, 2004, p. 173). According to the literature reviews, MEQs remain a current yet controversial topic in higher education research and practice with many participants questioning the validity and reliability of MEQ results (Ory, 2001), and research on SETs has thus far failed to provide clear answers to the validity and reliability of SETs (Spooren et al., 2013). From the evaluation of University X's MEQ statements, unclear questions imply that University MEQs as unreliable as an evaluation tool.

Strength/weakness
University MEQs allow students to provide their opinions about the course, which is one of the strengths of the University MEQs. However, the main weakness concerns the unclear purpose of university MEQ statements with vague and unclear wordings, which resulted in various students' interpretations and answers.

Suggestions
There are three suggestions. Firstly, MEQ statements should be clearly stated for students to understand the value of participating. It is suggested a staff-student partnership to agree on the purpose of the university MEQs and co-design a revised instrument, which may help to meet the stated purpose. Secondly, students' educational cultural differences should be taken into consideration in conducting MEQs. While course evaluation is well-established practice in North America (Perry, 1990;Perry and Smart, 1997;Marsh and Dunkin, 1992;McKeachie, 1990), such practice particularly in Asia is uncommon (Marsh, 1987;Watkins, 1994). In Asia, students are not expected to question their tutors' knowledge and authority and they are taught to respect, not criticise, their tutors (Ting, 2000).
Finally, university MEQs and academic promotion are seemingly related, relying on university MEQs as a sole and the main determinant for tutors' career progression needs to be cautious. University MEQs should be used in conjunction with other quantitative evidence such as class average attendance rate, average, maximum and minimum marks as well as qualitative response analysis which would help build a more accurate overall picture of the class and help to reduce the bias effects outlined above. Further, university MEQs and the above-mentioned teaching-related data should be provided to promotion panels to avoid the cherry picking of comments or data by applicants.

Conclusions
To answer RQ1, university MEQs have three purposes: (1) customer satisfaction for the students; (2) accountability for university authorities; and (3) teaching and academic promotions for teaching staff. To answer RQ2, University MEQs at University X considered unreliable. As the majority of MEQ statements are unclear in their purpose, it is possible to say that the results of MEQs are unreliable.
MEQs remain a current yet controversial topic in higher education research and practice with many participants questioning the validity and reliability of MEQ results (Ory, 2001), and research on SETs has thus far failed to provide clear answers to the validity of SETs (Spooren et al., 2013). This suggests that a gap exists between the institutional reality and the literature reviews. The fact that unreliable MEQs are still heavily relied upon in academic promotion highlights issues in academic reality.
In introduction, the question was asked what the recent shift and strong emphasis on lecturers' scholarship activities affects in the introduction. The answer may be concluded that this shift may be welcoming movement to steer away from heavily relying on unreliable University MEQ results for academic promotion.