Measuring preservice science teachers’ performance on engineering design process tasks: implications for fostering STEM education

Abstract

Purpose

The purpose of the current study is to assess Omani teachers’ performance on tasks related to the stages of engineering design. To achieve this, data from an engineering design test was used, and demographic variables that are correlated with this performance were identified.

Design/methodology/approach

This descriptive study employed a cross-sectional design and the collection of quantitative data. A sample of preservice science teachers from Sultan Qaboos University (SQU) (n = 70) participated in this study.

Findings

Findings showed low and moderate levels of proficiency related to the stages of engineering design. Differences between males and females in terms of performance on engineering design tasks were found, with females scoring higher overall on the assessment. Biology preservice teachers scored higher than teachers from the other two majors (physics and chemistry) in two subscales. There were also differences between teachers studying in the Bachelor of Science (BSc) program and the teacher qualification diploma (TQD) program.

Originality/value

This study provides an overview, in an Arab setting, of preservice science teachers’ proficiency with engineering design process (EDP) tasks. It is hoped that the results may lead to improved instruction in science teacher training programs in similar contexts. Additionally, this research demonstrates how EDP competency relates to preservice teacher gender, major and preparation program. Findings from this study will contribute to the growing body of research investigating the strengths and shortcomings of teacher education programs in relation to science, technology, engineering and mathematics (STEM) education.

Citation

Shahat, M.A., Al-Balushi, S.M. and Al-Amri, M. (2024), "Measuring preservice science teachers’ performance on engineering design process tasks: implications for fostering STEM education", Arab Gulf Journal of Scientific Research, Vol. 42 No. 2, pp. 259-279. https://doi.org/10.1108/AGJSR-12-2022-0277

Introduction

Over the past few decades, significant changes have been made to the Omani educational system and new science and math curricula created by Cambridge University are being implemented as part of these reforms. Likewise, interdisciplinary project initiatives by Rolls-Royce United Kingdom (UK) have been introduced that incorporate practical science, technology, engineering and mathematics (STEM) education programs. (Oman Educational portal, 2019). These reforms have come about due to the realization that Omani students were not reaching their potential academically. For instance, Omani students scored an average of 457 on the 2019 trends in international mathematics and science study (TIMSS) test, which was much lower than the international average of 500 (Mullis et al., 2020). In addition, according to TIMSS 2019, Omani students’ views toward science were relatively negative, with an average of 454, which is significantly lower than the global average of 524. According to Shahat, Ambusaidi, and Treagust (2022), teaching methods and student effort both contribute to the poor performance in science by Omani students. This is backed up by another study in the same context, which concluded that the methods used to teach students made a significant difference in their accomplishments (Shahat, Al-Balushi, & Al-Amri, 2022). If we combine this evidence with the finding that preservice instruction has a large impact on teachers’ competence and classroom practices (Shahat, Ohle, & Fischer, 2017), then it seems clear that teacher education programs are of vital importance.

Surveys of the literature have concluded that students exposed to practical STEM education in school acquire tools crucial for any future academic endeavor (Papadakis et al., 2021). This fits in well with the Omani government’s current vision for the education system, which places a strong emphasis on creating learning settings that are well organized and give each student high-quality instruction (Oman-2040, 2020). According to Zaher and Damaj (2018), the idea of STEM education has gained traction throughout the world in recent years and applying the next generation science standards (NGSS) to the integration of STEM education has received a lot of attention recently, especially in Oman (Shahat, Al-Balushi et al., 2022). The Ministry of Education in Oman launched the STEM Oman education program in 2018, which aims to add engaging practical STEM activities to science courses (Oman Educational portal, 2019). The key objectives of STEM Oman are to increase learning enjoyment, maximize teaching effectiveness and ensure the compatibility of science courses with the requirements of Omani society. However, because the recently implemented science curriculum lacks integration, it is important to not only offer a STEM-based curriculum, but also train science teachers to implement it cohesively. According to Haatainen, Turkka, and Aksela (2021), teachers’ opinions of integrated science teaching are influenced by their knowledge and experience in using integrated activities. A recent study in Greece concluded that although teachers and preservice teachers knew about the science, technology, engineering, arts and mathematics [STEAM] approach, only a few had experience in implementing it (Ampartzaki, Kalogiannakis, Papadakis, & Giannakou, 2022). Therefore, it is crucial to enhance Omani preservice teachers’ STEM skills by instructing them in the use of effective teaching methods. In order for this to happen, theory-based practical school training must be incorporated into education training programs to give preservice science instructors the required competence to deliver effective STEM education. In addition, research must be carried out to measure the impact of any training on student performance (Oman-2040, 2020). To accomplish the goals of the Oman STEM initiative, research from Mesutoglu and Baran (2020) in Turkey must be taken into consideration – they found that science instructors’ dedication to their work and an ability to integrate a methodical scientific approach into their STEM instruction were essential. Generally speaking, increased teaching knowledge improves a teachers’ performance in the classroom (Bandura, 1989; Shahat, Ambusaidi, & AlBahri, 2022); therefore, it is essential to address teacher training to ensure new teachers understand the methodological principles of the STEM approach and are flexible and skilled enough to deal with evolving classroom situations (Ampartzaki et al., 2022).

There is a need to improve the relatively poor science proficiency of Omani students, as reported by the TIMSS 2019 report (Ohle-Peters, Shahat, & Ambusaidi, 2022). Conducting in-depth research on pre- and in-service science teachers’ competency is one strategy to achieve this (Shahat, Al-Balushi et al., 2022). The fact that engineering design is not yet standard practice in many science classrooms in Oman is at least partially due to science teachers lacking guidance or training in designing and constructing physical products during experimental investigations (Al-Balushi, Al-Harthi, & Shahat, 2022). Some science teachers in Oman find it very challenging to teach courses which require the integration of STEM. Hence, it is imperative to try and uncover preservice teachers’ competency with the engineering design skills underpinning STEM education (Shahat, Ambusaidi, & Treagust, 2022). Despite a thorough search, no recent studies can be found from Arab countries that explore the proficiency of preservice science teachers with engineering design process (EDP) tasks. Thus, due to the usefulness of these processes in STEM instruction, this study aims to be the first to investigate preservice science teachers’ proficiency with EDP tasks. It will attempt to provide an overview of the current state of preservice teacher education in Arab contexts, and it may lead to improved instruction in science teacher training programs. This research will also show the relationship between proficiency with EDP tasks and demographic variables such as preservice teacher gender, major and preparation program. Underpinning this study is the idea that improved teacher education results in improved academic outcomes for students, particularly in terms of STEM programs in the future, which is of great importance for Oman.

In the following section we present an overview of the science teacher preparation programs in Oman.

Science teacher preparation in Oman

There are two types of teacher education programs in Oman operating in various public and private tertiary institutions around the country.

Bachelor program (BSc)

In order to teach biology, physics and chemistry in upper secondary schools and general science in lower secondary schools, BSc program graduates must possess a solid scientific background. In addition to this science grounding, BSc program students also receive training in teaching methodologies and culture. The SQU bachelor’s degree program for science teachers is intended to be completed in four years (8 semesters) (Shahat, Al-Balushi et al., 2022).

At SQU, the specialized science component makes up 60% of the program’s total credit hours, the methodologies component makes up 30%, and a cultural component makes up 10% (Public and Private universities in Oman, 2021). Students studying in this program take classes at the College of Science that are focused on various subjects and give candidates specialized scientific knowledge and a thorough appreciation of the inquiry-based character of science (Al-Balushi et al., 2022).

There is cooperation among faculties at SQU in order to teach trainee students the integrated Cambridge science curriculum which is currently implemented in Oman schools. These faculties offer courses focused on science, mathematics, technology and psychology, and on the foundation and leadership role of education. During the preparation period of the program, the student-teacher participates in field training through which they can test the effectiveness of the teaching skills they have learned (Shahat, Al-Balushi et al., 2022).

The Council for the Accreditation of Educator Preparation (CAEP) recognized and accredited SQU’s BSc in science education course in 2016. This accreditation provides proof that the science teacher training program adheres to high standards. Observers have also commented on the improving quality of science teaching in Omani schools by Bachelor of Science education graduates (Shahat, Al-Balushi et al., 2022).

Teacher qualification diploma (TQD)

Omani graduates with a BSc in science from an arts, scientific or technological institution may enroll in the teacher qualification diploma (TQD), a two-semester program designed to prepare science degree holders for teaching. This program focuses on helping student teachers learn about pedagogy and gain practical classroom experience through school placements in public schools (Shahat, Al-Balushi et al., 2022).

Theoretical background

The theoretical background presented here includes the EDP and its various conceptualizations, as well as an overview of relevant studies and their connection to the present study.

Engineering design

Engineering design can be defined in the present study as “an activity that involves the design of a physical product that solves a human problem” (Marulcu & Barnett, 2013, p. 1828). Each stage of the EDP reveals details about the problem and potential solutions. By actively going through the stages in the EDP, it is thought that learners can acquire methods of tackling problems and ways of thinking which can be applied in a wide variety of challenging and complex situations. In education, Mangold and Robinson (2013) described the EDP as an iterative decision-making process in which basic science, technology, math and engineering (STEM) ideas are applied to generate optimal solutions to meet a stated aim. Educators and researchers worldwide have promoted the integration of STEM in education, claiming it to be a catalyst for driving the global economy (Zainuddin & Iksan, 2019). The STEM approach is frequently combined with other approaches or learning models such as STEM project-based learning (Nurtanto, Pardjono, Widarto, & Ramdani, 2020), STEM problem-based learning (Tawfik, Trueman, & Lorz, 2014) and STEM through the EDP (Nurtanto et al., 2020; Sulaeman, Triwulandari, & Syam, 2022; Widiastuti, Oktavia, Lukad, & Sutrisno, 2022; Winarno et al., 2020). The EDP is a strategy for teaching STEM, which places a strong emphasis on STEM integration (Yu, Wu, & Fan, 2020). STEM teaching and learning that uses the EDP begins with introducing an open-ended design problem to provide realistic challenges while also giving students flexibility and choice (Hafiz & Ayop, 2019).

Those involved with real-world engineering design must deal with a wide variety of tasks in a wide range of situations. According to previous studies, including Shahat, Al-Balushi et al. (2022), the main engineering design tasks can be grouped into the following types: origin of the task, organization, novelty, batch size, branch, complexity and goals. To deal with the complex processes involved, these tasks must be approached in an organized fashion rather than relying merely on intuition. To this end, in engineering design, solving a problem usually necessitates students developing models. The various systematic EDP models have gradually evolved to include a common foundation and now their main differences are minor variations (Avsec & Szewczyk-Zakrzewska, 2018; Shahat, Al-Balushi et al., 2022; Capobianco, DeLisi, & Radloff, 2018; Kaya et al., 2017; Mourtos, 2012; Watkins et al., 2018; Yu et al., 2020). A model can take many forms, including graphical, physical or mathematical representations of the essential features of a system or process to aid in engineering design (Hafiz & Ayop, 2019; Shahali, Halim, Rasul, Osman, & Zulkifeli, 2017; Uzel & Bilici, 2022).

EDP-based pedagogy can enable integrated STEM education and support collaboration and professional development among instructors. Engineering design tasks can also give students and their teachers the chance to see how ideas and procedures from various fields can be applied to design process tasks such as problem formation, predictions of prototype testing and analysis of prototype testing (Donna, 2012). Additionally, the use of these tasks gives educators from a variety of STEM fields a chance to support students’ learning of ideas and procedures from other fields.

In order to support the learning of STEM concepts within and across subjects, Donna (2012) produced a research-based professional development model that promotes engineering design. The first phase of the model is the exploration of prior knowledge related to engineering and relationships between different subjects. The second phase is the development of a basic knowledge of engineering. The third phase involves engagement with a cooperative engineering design activity. The fourth phase consists of reflection on the activity as both learners and STEM educators. The fifth phase is the extension of knowledge and connections between domains. And the last phase of the model is the continuation of work within professional learning communities.

The goal of using engineering design is to motivate students to engage in practical engineering activities that put their math and science skills to use; through the process, students should come to realize that engineering is more than just building things with their hands. Hynes, Portsmore, Dare, Milto, and Rogers (2011) developed a nine-phase model for incorporating engineering design into STEM education. The engineering design stages Haynes et al. described, which were adapted and used in developing the questionnaire in the current study, were (1) identify a need or problem; (2) research the need or problem; (3) develop possible solutions; (4) select the best possible solution; (5) construct a prototype; (6), test and evaluate solution; (7) communicate the solution; (8) redesign; and (9) completion decision. Marulcu and Barnett (2013) identified similar engineering design steps that we considered when developing our questionnaire: (1) identifying a problem; (2) researching possible solutions; (3) picking the best solution; (4) building a prototype; (5) testing the prototype and (6) repeating any steps needed to improve the design. When students follow the steps of engineering design, they are not only exposed to practical design scenarios, but also, it is hoped, their critical thinking, mathematical inquiry and computational thinking about the evidence are strengthened (Shahat, Al-Balushi et al., 2022).

The engineering design approach to identifying and solving problems, according to Mangold and Robinson (2013), is characterized by (1) its highly iterative nature; (2) the possibility of open-ended solutions; (3) the creation of a meaningful context for learning scientific, technological, engineering and mathematical concepts; and (4) the stimulation of students to consider systems, modeling and analysis. They claim it provides a defined framework for designing and organizing courses; offers an appropriate platform for combining mathematics, science, technology and engineering; and provides students with practical challenges that reflect real-world tasks. Zainuddin and Iksan (2019) found that individually and collaboratively, students were able to design innovative solutions when engaged with engineering design activities.

The engineering design process in science education

Previous research on using the EDP in science education has used a range of quantitative, mixed and qualitative methodologies. Considerable variation has been found in descriptions of the stages of EDP. Capobianco et al. (2018) labeled the engineering design stages used by elementary science teachers as identify the problem, develop a plan, create, test, communicate results, improve and retest. However, Watkins et al. (2018) argued that the engineering design stages teachers need to know are designing, building, testing, redesigning, rebuilding and retesting. In a qualitative case study, Mesutoglu and Baran (2020) looked at how middle school science teachers’ understanding of the EDP progressed in Turkey. The study outlined five major steps in the EDP: define the problem, plan possible solutions, choose the possible solution, design and test. Their findings indicated that a complete understanding of engineering design requires an awareness of (1) the iterative nature of the EDP; (2) the importance of redesigning and communication; and (3) the benefit of the EDP to society.

Following a four-step EDP - learn, plan, teach and reflect - Wendell, Swenson, and Dalvi (2019) conducted a comparative case study in the United States of America (USA) of two novice teachers who taught engineering design during a four-week program for new elementary teachers. Despite having very similar backgrounds, the two teachers interpreted the purposes of engineering design learning and teaching in markedly different ways. Wendell et al. (2019) suggested that to avoid confusion teacher trainers should assist teachers to frame the use of engineering design as a knowledge-building tool by (1) engaging in explicit conversations about epistemology; (2) showing teachers strategies for approaching problems and (3) helping teachers design tasks that intentionally encourage critical thinking. Lie, Aranda, Guzey, and Moore (2021) also used a qualitative approach; however, they focused on investigating students’ conceptions of design before and after participating in a design-based science unit. They examined students’ conversations throughout the teaching of the unit and conducted interviews after it was completed. Overall, it was found that students demonstrated reasonable design skills throughout the process. Furthermore, the analysis of the group discussions students had during a design challenge revealed that students’ thoughts on design were complex and varied.

According to Chen, Huang, and Wu (2021), preservice teachers who had experience teaching STEM subjects exhibited an interest in STEM and STEM-related activities. Geng, Jong, and Chai (2019) examined teachers’ concerns regarding STEM education and found that for teachers to implement STEM education in the classroom successfully they require professional development, pedagogical support and curriculum resources. According to a recent study by Shume, Bowen, Altimus, and Kallmeyer (2022), teacher training that is focused on research and implementing the EDP has an impact on teachers by helping them acquire genuine conceptual knowledge of engineering design tasks and by inspiring them to come up with new problems for their students to solve.

The approach of teaching science using engineering design is a useful method for leveraging students’ creativity and applying it to real world issues. For example, Wendell et al. (2019) found that engineering design-based instruction can help students think creatively and in an interdisciplinary manner to solve problems. McFadden and Roehrig (2019) highlight the importance of the EDP in helping students learn to draw designs and in assisting them to use these drawings to help conceptualize a problem during an engineering design challenge.

In conclusion, STEM education is one strategy that can help prepare students for the 21st century. The EDP approach can be used in STEM teaching and can link all the STEM subjects. When introducing engineering design as a teaching approach to preservice science teachers, this research may help guide this instruction as it sheds light on the ability of preservice science teachers to complete the stages of the EDP.

Research aims and questions

The aims of this study are twofold: (1) investigate preservice science teachers’ competency regarding EDP tasks; and (2) examine the effect of preservice science teachers’ gender, major and preparation program on the performance on tasks related to the EDP. The main research questions (RQs) are as follows:

RQ1.

How do preservice science teachers in the Sultanate of Oman perform on an EDP challenge?

RQ2.

Do either gender, major or preparation program influence preservice science teachers’ performances on EDP tasks?

Methodology

Design

This was a descriptive cross-sectional study (Jason & Glenwick, 2016) that utilized quantitative data. Consideration was given to pertinent student teacher demographic variables, i.e. gender, major and preparation program. Each participant was asked to complete two engineering design challenges based on their major, with each challenge split up into eight stages related to the EDP.

Participants

In this study, a sample of 70 student teachers from the College of Education at SQU in Oman (M_age: 22.80) took part. It included student teachers studying the Bachelor of Science (BSc) for education (n = 31, 70.45% of the total enrolled student teachers) and the TQD (n = 39, 70.90% of the total enrolled student teachers (see Table 1). All preservice science teachers participated voluntarily. The sample was chosen with formal consent from SQU management. The study was conducted in the 2020/2021 academic year.

Instrument

Fair test to measure engineering design ability (FTMEDA)

The researchers adapted the engineering design steps that were used by Marulcu and Barnett (2013), and Hynes et al. (2011) to measure engineering design ability. This prior work was believed to be a reasonable foundation for building the FTMEDA. Physics, chemistry and biology majors from the BSc and TQD programs were selected and given scientific challenges relating to their majors. The students worked individually and wrote their answers on exam papers. Responses relating to the following stages were then analyzed: problem identification, solution investigation, planning, production, testing, communication, production and improvement (adapted from Hynes et al., 2011; Marulcu and Barnett, 2013). For the design challenge, students were asked to do the following (the name of the stage is in parentheses):

1. Write a question that identifies the problem (Identifying the problem).

2. Clarify the dimensions of the problem and the factors associated with it (e.g. scientific concepts/previous solutions) (investigating solutions)

3. Write down all possible solutions (Brainstorming – imagine)

4. Select the most appropriate answer among the solutions offered. Explain why? (Planning)

5. Design a model to solve the problem (e.g. sequences, drawing) (Production)

6. Clarify the design testing mechanism. (Test)

7. Explain how the product will be presented. (Communication)

8. Develop a product development plan. (Improvement)

Descriptions of the highest scoring criteria for each challenge stage are as follows: (1) Identifying the problem: clearly defines the problem with the definition of the criteria and determinants of the solution; (2) Investigating Solutions: identifies all scientific concepts associated with the problem; (3) Brainstorming – imagine: writes at least three or more solutions; (4) Planning: makes it clear that the choice of the appropriate solution is in accordance with the criteria and determinants. (5) Production: designs a model to solve the problem that includes all engineering design steps; (6) Test: Designs a design testing mechanism according to specific criteria; (7) Communication: ensures all the following elements are in the presentation: identify the problem/steps of design production/testing/and uses suitable methods to illustrate the presentation; (8) improvement: develops a development plan based on design test results and a discussion of the design with other groups during the presentation.

Six inquiry challenges were developed for this study (2 for biology majors, 2 for chemistry majors and 2 for physics majors) (Table 2). All these challenges are new and were developed expressly for this study. The challenges presented to the three majors are outlined below.

Chemistry:

1. Design a method for producing solid biofuels as an alternative to industrial coal, using organic materials available at home (e.g. coffee residues, olive oil residues, animal dung after fermentation etc.) It needs to burn for five minutes.

2. A chocolate manufacturer has contracted you to develop chocolate coffee cups as an alternative to plastic cups. One cup must be able to withstand a temperature of 70°C and have a total weight not exceeding 500 g.

Physics:

1. A confectionery factory has a long roof and wants to create a huge marshmallow monument in the form of a tower with a giant marshmallow piece on top (at the highest point). It should be installed for as long as possible and the entire construction (tower and marshmallow) should be made with only 30 wooden sticks of equal size.

2. You have been contracted to develop a catapult as a marketing tool for a cutlery (spoons, forks and knives) manufacturer. You must work to develop a stand-alone spoon catapult and launch a projectile a distance of half a meter. Tools from the environment may be used.

Biology:

1. A major commercial complex would like you to design a tank for housing fish from the Amazon River. The environment within the tank should be suitable for Amazon River fish, and the capacity should not exceed 100 cubic meters.

2. A bakery is facing some problems related to the quality of its baked goods. They found that the most important reason for the low quality was that the bread has not been fermenting properly. You have been chosen to help them find a solution to this problem using specific materials and tools.

Content validity was considered by asking five experts in Oman to assess the instrument used while considering the relevant literature. The experts were asked to judge the accuracy of the developed items and their suitability for judging each stage of the EDP; they were also asked whether they thought the six challenges were of equal difficulty. Cohen Kappa was acceptable, with values between 0.82 and 0.90. To determine each participant’s level of engineering design ability, a rubric was designed to allow grading of each challenge stage. The inter-rater reliability of the instrument was determined by six science education post-docs (two physics, two chemistry and two biology majors) using the six challenges. Cohen’s Kappa values of 0.80 to 0.88 were obtained, meaning there was almost perfect agreement (Field, 2013). Table 3 gives an example of the grading process for a physics challenge (see Table 4).

Data analysis

This study incorporated the inductive method, which initially involves looking for patterns in observations. To explain these patterns, theories are developed through a sequence of hypotheses. In this way, theory can be derived from data representing participants’ lived experiences (Leach Sankofa, 2022). Participants’ responses were grouped according to the stages in the engineering design challenge (Table 3). Cohen’s kappa (k) was used to assess the inter-rater agreement between the evaluators (Field, 2013). Items were coded as follows: correct answer = 3, mostly correct = 2, partially correct answer = 1. In order to answer the RQs, mean scores were calculated. In order to address RQ1, the mean value of each item and stage was determined and then categorized into one of three performance levels (high performance, moderate performance and low performance), as shown in Table 5. The range of the means was determined using the following formula (cf. Shahat, Ambusaidi, Al Busaidi, & Al Qulhati, 2022):

Length of interval =Highest value(5)Lowest value(1)÷number of options=(3−1) ÷3 =0.67

The value (0.67) was added to the minimum value, which was 1. Following this, the result (1.67) was added to the interval length (0.67) and this process was repeated until the maximum value of 3.0 was reached (see Table 2). To answer RQ2, the impact of gender, major and preparation program was examined using multivariate analyses of variance (MANOVA).

Findings

Descriptive statistics

Good levels of reliability were obtained for the eight stages of the FTMEDA for all three majors (Cronbach’s Alpha ≥0.65) (Griethuijsen et al., 2014).

Response to RQ 1: how well do preservice science teachers in the Sultanate of Oman perform on an engineering design process challenge?

Based on the two challenges provided to the participants, the responses from the FTMEDA indicated that the teachers have low or moderate abilities related to EDPs (as shown in Table 5). For example, for challenge 1 the mean total score in physics was 1.58 (standard deviation (SD) = 0.25); for chemistry M = 1.54 (SD = 0.33); and for biology M = 1.64 (SD = 0.32). Similar results were obtained for the challenge two total scores. Table 6 illustrates that the preservice teachers scored either low or moderate in all eight stages of the FTMEDA.

Response to RQ 2: do either gender, major or preparation program influence preservice science teachers’ performances on engineering design process tasks?

RQ2a: gender differences

The results (Table 6) show statistically significant gender differences between the mean scores for teachers on the investigating solutions, production and testing, subscales and the full FTMEDA scale. For example, the MANOVA results, F_{(1, 6)} = 8.37, p > 0.05, revealed that female teachers’ abilities related to the EDP tasks were significantly higher than those of male preservice teachers.

RQ2b: Major differences

Analysis of variance (see Table 7) showed a significant main effect of major on three subscales of the FTMEDA; these were identify problem F_{(2, 67)} = 3.87 (p < 0.05), planning, F_{(2, 67)} = 4.44 (p < 0.05), and communicate F_{(2, 67)} = 7.21 (p < 0.05). For identify the problem, post hoc analyses using Tukey’s HSD (honestly significant difference) indicated that preservice physics teachers did better on the challenges than chemistry and biology preservice teachers. Regarding planning and communication, post hoc analyses indicated that biology preservice teachers performed better than students from the other two majors.

RQ2c: Preparation program

The MANOVA results reveal a significant effect of the preparation program on preservice teachers’ abilities related to the EDP tasks in one or more subscales, Wilk’s Lambda = 0.84, F = 3.30, p < 0.05. Table 8 shows a significant effect of the preparation program on two scales of the FTMEDA, identifying the problem, F_{(1, 68)} = 3.77, p < 0.05, and possible solutions, F_{(1, 68)} = 7.93, p < 0.01. However, no significant effect of preparation program was found for the rest of the subscales or the whole scale, p > 0.05.

Discussion and conclusion

The purpose of this study was to investigate SQU preservice science teachers’ engineering design abilities. For RQ1, the findings indicated that the performances of preservice teachers on the engineering design tasks were low or moderate. This finding is in line with other recent Omani studies that have showed that most teachers believe they need more practice related to teaching and implementing EDP in the classroom (e.g. Alkharusi, 2020). However, the findings contrast with a study conducted in Oman by Elayyan and Al-Shizawi (2019), their results showed positive perceptions among science teachers towards integrating STEM in teaching science. Bandura’s (1989) social cognitive theory asserts that teachers’ self-efficacy beliefs and perceptions influence their teaching performance and sense of professionalism in classroom situations. We argue that the low and moderate levels of performance related to the EDP challenges in this study were influenced by the learning experiences the participants had during their preparation programs; specialized subject courses in Oman teacher preparation programs are studied in isolation from each other and are not combined with pedagogy courses. Therefore, there seems to be little integration across STEM subjects, and this is reflected in the current course evaluation processes (such as the National Science Teaching Association (NSTA)), which are very much based on individual subjects. A recent study by Elayyan and Al-Mazroi (2020), revealed three types of obstacles, from a teachers’ point of view, that limit the implementation of the STEM approach in Oman science education; these were related to content, the learning environment and teachers. A more recent study by Ampartzaki et al. (2022) found that the major difficulties student teachers and teachers faced in implementing STEAM education effectively in the classroom were related to understanding the methodological principles of this approach and a lack of educational resources. In general, previous research has reported that acquiring sufficient knowledge about engineering design is an essential factor for teaching science using engineering design (Hammack & Ivey, 2017). The teachers in the study believed that (1) the curriculum content in school was not presented as fully integrated subjects; (2) the learning environment made implementing the activities very challenging; and (3) teachers were not entirely prepared regarding STEM education requirements. Hammack and Ivey concluded that this type of education has material, training and logistical challenges that Oman should seek to remedy.

Other researchers have concluded that teacher preparation programs need to consider more comprehensive courses on topics such as communication, technology, curriculum design, the EDP in math and science education, transdisciplinary learning approaches and the implementation of STEM in the classroom (Shahat and Al-Amri, n.d.). Shume et al. (2022) suggested that having student teachers conduct their own research regarding STEM can positively impact teacher training programs by helping student teachers develop a more authentic conceptual understanding of engineering design tasks, and it can help them create original problem-solving challenges for students in the classroom. Also, more consideration must be given regarding integrating STEM subjects through reflection on (1) the instruction received during their training; (2) school placement experiences and (3) teaching during microteaching (College of Education at SQU, 2021). In addition, there is a need to reduce the number of specialized courses and offer a more integrated range of subjects. Finally, there is a need to focus more on the practical rather than theoretical aspects of science teaching. Providing more activities such as engineering design tasks that incorporate realistic problem solving and the fostering of creative thinking are therefore essential, and prototype lessons should be designed to aid inexperienced teachers.

This study also showed that female participants had significantly better engineering design skills than male participants (RQ2a). We argue that there was a gender difference because female teachers are typically more involved in their courses, as found by national studies conducted in Oman (e.g. Shahat, Ambusaidi, Al Busaidi et al., 2022). They have also been found to be more eager to participate in a wide variety of in-service training courses, including ones on EDP, to hone their teaching abilities. This argument aligns with previous research, which showed that having a variety of experiences with engineering design positively influenced science teachers’ ability to utilize these approaches in the classroom (e.g. Chen et al., 2021). Boys and girls are taught in the same classes in cycle 1 (grades 1–4), and these grades typically have female teachers. In cycle 2 (grades 5 to 10) and post-basic education (grades 11 to 12), however, male and female students are taught in separate schools, and the teachers are the same gender as the students. The students of female teachers provide further evidence of a gender effect, with female students outperforming male students in tests such as the TIMSS in grade 8 (Shahat, Al-Balushi et al., 2022).

Regarding differences in major (RQ2b), the results indicated that for identifying the problem, physics preservice teachers scored higher than chemistry and biology preservice teachers. Regarding planning and communication, biology preservice teachers scored higher than in the other two majors. The study’s finding is in agreement with previous studies demonstrating the significant impact of major on teachers’ perceptions of STEM teaching practices in the classroom (e.g. Srikoom, Hanuscin, & Faikhamta, 2017). One possible explanation for why physics majors performed well on identifying the problem could be that physics majors have several opportunities to solve problems and challenges in their major courses (i.e. General Physics I, General Physics II, Physics III, Methods of Teaching Science I, Methods of Teaching Science II) and this may improve their performance in identifying the problem. Regarding the performance of biology majors, we argue that SQU offers all students, and in particular biology students, several opportunities to plan their work and communicate what they found in their assignments by using figures, diagrams, infographics and text and picture integration. This integration of elements to effectively communicate ideas is an essential characteristic of EDPs (Roehrig, Dare, Ring-Whalen, & Wieselmann, 2021). A recent nationwide study in Oman revealed that teachers had high levels of understanding of text-picture integration (Shahat, Ohle-Peters, & Ambusaidi, 2022).

The results also indicated a substantial impact of the preparation program (RQ2c) on engineering design task performance. The use of an engineering design approach has not been an explicit pedagogical theme found in the BSc or TQD materials in Oman. The BSc and TQD professors did, however, permit the student teachers to put what they had learned about engineering design procedures to use and adapt the textbook exercises to fit the engineering design cycle (Shahat & Al-Amri, n.d.). Notably, BSc students’ performed better in the identify the problem stage and TQD students’ performed better in the possible solutions stage. The scientific teaching methods course lasts two semesters in the BSc degree; however, the scientific teaching methods course in the TQD programme only lasts for one semester, which restricts the knowledge and abilities that these students can acquire. Students in the teaching methods course examine an issue in education and then, while taking into account the processes of scientific inquiry, offer ideas and possible solutions. Also, the teaching methods course (and field training courses) exposes students to EDPs (Ambusaidi, Shahat, & Al Musawi, 2022). This integration of theory and practice is essential to a STEM education and engineering design-based curriculum (Shahat & Al-Amri, n.d.). A study by Shahat, Al-Balushi et al. (2022) on Omani preservice teachers’ awareness regarding teaching science using the EDP approach, found that BSc participants were given several opportunities to solve challenging science problems in their teaching methods courses (I & II), which may have improved student performance in identifying the problem. However, it was found that students in the TQD program had more opportunities to provide solutions to scientific problems through their assigned projects at the College of Science.

In conclusion, the results presented here contribute to the body of research that sheds light on the impact of science teacher preparation programs on teaching approaches and the performance of future teachers. The results apply to comparable circumstances in other colleges and universities, not only in Oman and Arab nations but also globally. The findings should help provide an overview of the current issues facing science teacher training programs. More specifically, it demonstrated a connection between competency with EDP tasks and factors such as the student teachers’ gender, major and preparation program. Hopefully, reading this article will help aspiring science teachers learn the benefits of the EDP in science education and push them to champion its use. This will consequently lead to students actively engaging in and learning from STEM programs, which is crucial for Oman.

Challenges

The main issue faced by student teachers during the challenge phase of this study was mostly related to a perceived lack of time to solve the challenges presented to them. This, of course, would also likely be an issue when teaching STEM to students in the classroom as substantial time is required to implement STEM lessons correctly. Another issue that was faced in carrying out this study was finding a suitable time for student teachers from different majors and programs to meet and work on the study challenges. A time that suited all students was chosen based on their schedules.

Implications for teaching and research

The principal educational contribution of this study is the descriptions of the stages of the EDP used in the FTMEDA: problem identification, solution investigation, planning, production, testing, communication and improvement; and also, the grading scales covering three basic dimensions of instructional quality. These eight stages of the EDP could be used as part of a diagnostic measure to identify strengths and weaknesses related to teaching STEM subjects in the classroom. Such a measure might help education decision makers when conducting needs analyses and trainings for science teachers while considering EDPs. Teachers could also use peer evaluation to investigate the strengths and weaknesses of each other’s performances when teaching science using the engineering design approach. These reports could then be used to document best practices for teaching STEM in classrooms.

Another implication of this study for the growing body of research in STEM education is that instructors in the BSc and TQD programs in Oman may need to include a teaching model for an engineering design approach. We recommend explicitly incorporating the engineering design model into the teaching of science courses to enable student teachers to teach STEM subjects effectively. In addition, both genders of preservice science teachers need more professional training regarding EDP tasks and may require additional training during microteaching and in-school practicums to develop their competence in STEM education settings. Another of this study’s important results is that science teachers in the TQD and BSc programs may require additional training to increase their understanding of the EDP stages and how they can impact classroom practice (Lavidas, Apostolou, & Papadakis, 2022). This is necessary to promote effective classroom practices and, as a result, ensure that high-quality STEM instruction is occurring in Omani science classrooms.

Yet another contribution of this study to the field is the further validation of the established FTMEDA instrument. It demonstrated good psychometric parameters for use in Oman and could be used in other contexts to measure proficiency with EDP tasks and assess STEM education in science teacher education programs. We recommend conducting qualitative studies using observations and interviews and further quantitative studies using questionnaires and other types of data collection methods to further assess preservice teachers’ competence in STEM education and uncover difficulties teachers are facing in implementing STEM-based instruction. It is also essential to assess the influence of the various training opportunities on preservice teachers’ practices related to STEM education in schools. Possible further research could involve a training intervention study focused on instructing preservice science and mathematics teachers to utilize EDPs in the classroom and measuring its influence on students’ learning outcomes related to STEM.

Limitations

A cross-sectional research design, by its very nature, gathers data from a single point in time, meaning potential causality and directionality for any relationships cannot be inferred. Therefore, we recommend another study examining the impact of preservice teachers’ proficiency with the EDP on students’ academic performance. Furthermore, the purposive sampling method used in this study and the small sample size should be considered major limitations of this study. However, performance on these tasks may be representative of how preservice science teachers from other science teacher preparation programs in other colleges or institutions in Oman would perform. In future studies, we could improve the validity and reliability of FTMEDA by having a larger sample size (Field, 2013). For the current study, responses were only obtained for two challenges for each major due to a lack of resources; the validity and reliability of the test could be improved by using more challenges. Another limitation of the current study is related to the comparison of the different majors for RQ2b. Since students from each major received different EDP challenges from the other two majors, the differences found among the three majors could be due to the nature of the challenges they received. However, when the different challenges for the FTMEDA were designed, the authors used expert judgment to make sure that these challenges were similar in difficulty. Also, none of the experts who reviewed the instrument commented that the nature of the challenges would be an issue. We would recommend that future studies use tests such as the post-hoc t-test for estimating the similarity of challenges.

Table 1

Sample statistics

Variable		No
Major	Biology	31
	Physics	11
	Chemistry	28
Gender	Female	50
	Male	20
Programme	BSc	31
	TQD	39

Source(s): Authors work

Table 2

Range of mean values for each category of the scale

Level of performance	Range of mean value
High	2.36–3.00
Moderate	1.68–2.35
Low	1.00–1.67

Source(s): Authors work. Combined data from BSc and TQD programmes

Table 3

An example of the grading process for a physics challenge

A cutlery (spoons, forks and knives) manufacturer has been contracted to develop their cutlery. You must work to develop a stand-alone spoon catapult and launch a projectile a distance of half a meter. Tools from the environment may be used (using only the available tools: Plastic spoon, wide wooden sticks, rubber bands, adhesive tape, woolen ball)
Write a question that identifies the problem (identifying the problem)
1	2	3
1- Does not identify the problem correctly	Clearly identifies the problem while ignoring the criteria for the solution: (developing a stand-alone spoon catapult, firing a distance of half a meter) or omitting to mention the determinants	Clearly defines the problem with the definition of the criteria and determinants of the solution: (the development of a stand-alone spoon catapult, and to fire a distance of half a meter, using only the available tools: plastic spoon, wide wooden sticks, rubber bands, adhesive tape, woolen ball
2- Clarify the dimensions of the problem and the factors associated with it (scientific concepts/previous solutions/……) (Investigating solutions)
1	2	3
Only explains one scientific concept or mentions only one solution	Explains some of the scientific concepts associated with only one solution	Identifies all scientific concepts associated with the problem (momentum, anchor point, power arm, resistance arm, mass, weight) and explains through research a number of previous solutions (images, chart, etc)
3- Write down all possible solutions (brainstorming - imagine)
1	2	3
Writes only one solution	Writes two or fewer solutions	Writes at least three or more solutions
4- What is the most appropriate answer among the solutions offered? And why? (Planning)
1	2	3
Mentions a reason that has nothing to do with determinants or criteria	Makes it clear that choosing the right solution is only in accordance with the criteria	Makes it clear that the choice of the appropriate solution is in accordance with the criteria and determinants
5- Design a model to solve the problem (in steps/drawing/……) (Production)
1	2	3
Includes only one or two steps of engineering design	Includes most of the engineering design steps	Includes all engineering design steps: Identify the problem – investigate solutions – imagination – planning – production – testing – improvement – communication
6- Clarify the design testing mechanism. (Test)
1	2	3
Designs a testing mechanism without paying attention to standards	Designs a test mechanism without explicitly stating the criteria	Designs a design testing mechanism according to specific criteria Demonstrates the catapult testing mechanism produced according to the specified standard: stand-alone, releasing a distance of half a meter
7- Explain the mechanism of displaying the product. (Communication)
1	2	3
Mentions only one of the following elements: identifying the problem/design production steps/testing/and does not use suitable methods to illustrate the presentation	Ensures some of the following elements in the presentation: identify the problem/design production steps/test it/and uses suitable methods to illustrate the presentation	Ensures all the following elements are in the presentation: identify the problem/steps of design production/testing/and uses suitable methods to illustrate the presentation (programs – images – practical presentation…)
8- Develop a product development plan. (Improvement)
1	2	3
Develops a development plan without paying attention to - Design test results - Discuss with others while presenting results	The development plan is based on only one of the following - Design test results - Discussion of the design with other groups during the presentation	Develops a development plan based on - Design test results - Discussion of the design with other groups during the presentation

Source: Authors work

Table 4

Reliabilities of the components of the fair test

No.	Scale	Items	N (Ph/Ch/Bio)	Cronbach’s alpha (Ph/Ch/Bio)
1	Identify the Problem	2	11/28/31	0.59/0.79/0.77
2	investigating Solutions	2	11/28/31	0.70/0.81/0.76
3	Possible Solutions	2	11/28/31	0.68/0.80/0.77
4	Planning	2	11/28/31	0.62/0.79/0.77
5	Production	2	11/28/31	0.60/0.80/0.76
6	Testing	2	11/28/31	0.71/0.77/0.77
7	Communicate	2	11/28/31	0.69/0.78/0.79
8	Improvement	2	11/28/31	0.70/0.87/0.77

Source(s): Authors work. Combined data from BSc and TQD programs

Table 5

Descriptive statistics for subscales of FTMEDA

	Subscale	N	M (SD) level	M (SD)	M (SD)
		Ph/Ch/Bio	Ph	Ch	Bio
Challenge 1	Identify the Problem	11/28/31	2.23 (0.75) Moderate	2.05 (0.52) Moderate	2.02 (0.89) Moderate
	investigating Solutions	11/28/31	1.86 (0.77) Moderate	1.66(0.74) low	1.52 (0.67) low
	Possible Solutions	11/28/31	2.00 (0.00) Moderate	1.84 (0.37) Moderate	1.90 (0.29) Moderate
	Planning	11/28/31	1.77 (0.75) Moderate	1.63 (0.65) low	1.87 (0.82) Moderate
	Production	11/28/31	1.05 (0.21) low	1.14 (0.44) low	1.16 (0.45) low
	Testing	11/28/31	1.95 (0.78) Moderate	1.80 (0.94) Moderate	2.03 (0.76) Moderate
	Communicate	11/28/31	0.82 (0.59) low	1.02 (0.40) low	1.10 (0.29) low
	Improvement	11/28/31	1.00 (0.43) low	1.21 (0.70) low	1.50 (0.69) low
	Total Challenge 1	11/28/31	1.58 (0.25) low	1.54 (0.33) low	1.64 (0.32) low
Challenge 2	Identify the Problem	11/28/31	2.27 (0.77) Moderate	2.18 (0.74) Moderate	1.73 (0.89) Moderate
	investigating Solutions	11/28/31	1.41 (0.50) low	1.23 (0.60) low	1.19 (0.47) low
	Possible Solutions	11/28/31	1.41 (0.66) low	1.59 (0.63) low	1.74 (0.59) low
	Planning	11/28/31	1.05 (0.95) low	1.46 (0.91) low	1.79 (0.96) low
	Production	11/28/31	0.82 (0.39) low	1.07 (0.63) low	0.97 (0.47) low
	Testing	11/28/31	2.05 (0.95) Moderate	2.07 (1.09) Moderate	1.73 (1.05) low
	Communicate	11/28/31	0.55 (0.51) low	0.93 (0.46) low	0.92 (0.32) low
	Improvement	11/28/31	0.68(0.56) low	1.07 (0.63) low	1.06 (0.72) low
	Total Challenge 2	11/28/31	1.28(0.35) low	1.45 (0.45) low	1.39 (0.49) low

Source(s): Authors work. Combined data from BSc and TQD programs

Table 6

Means, standard deviations, and MANOVA results for the FTMEDA by gender

Scale	Gender	N	M	SD	F	df	p
Identifying the Problem	Female	50	2.08	0.65	2.75	1	0.09
	Male	20	1.88	0.57		68
Investigating Solutions	Female	50	1.50	0.52	6.67	1	0.01*
	Male	20	1.26	0.43		68
Possible Solutions	Female	50	1.77	0.36	0.20	1	0.65
	Male	20	1.73	0.42		68
Planning	Female	50	1.70	0.68	1.90	1	0.17
	Male	20	1.52	0.65		68
Production	Female	50	1.11	0.39	7.22	1	0.00*
	Male	20	0.92	0.33		68
Testing	Female	50	2.02	0.78	6.14	1	0.01*
	Male	20	1.66	0.76		68
Communicate	Female	50	0.97	0.37	2.67	1	0.10
	Male	20	0.86	0.35		68
Improvement	Female	50	1.17	0.55	0.19	1	0.66
	Male	20	1.12	0.55		68
Total Scale	Female	50	1.54	0.31	8.37	1	0.00*
	Male	20	1.37	0.31		68

Source(s): Authors work. Combined data from BSc and TQD programs, and combined data from the two challenges

Table 7

MANOVA results for the effect of major on preservice science teachers’ abilities related to engineering design processes

Scale	Major	N	M	SD	df	F	p
Identify the Problem	Biology	31	1.87	0.68	2 67	3.87	0.02*
	Chemistry	28	2.11	0.55
	Physics	11	2.25	0.63
Investigating Solutions	Biology	31	1.35	0.47	2 67	2.53	0.08
	Chemistry	28	1.44	0.51
	Physics	11	1.63	0.56
Possible Solutions	Biology	31	1.82	0.37	2 67	1.47	0.23
	Chemistry	28	1.71	0.40
	Physics	11	1.70	0.33
Planning	Biology	31	1.83	0.70	2 67	4.44	0.01*
	Chemistry	28	1.54	0.58
	Physics	11	1.40	0.73
Production	Biology	31	1.06	0.35	2 67	1.64	0.19
	Chemistry	28	1.10	0.45
	Physics	11	0.93	0.23
Testing	Biology	31	1.87	0.71	2 67	0.20	0.81
	Chemistry	28	1.93	0.89
	Physics	11	2.00	0.77
Communicate	Biology	31	1.00	0.21	2 67	7.21	0.00*
	Chemistry	28	0.97	0.40
	Physics	11	0.68	0.50
Improvement	Biology	31	1.28	0.54	2 67	5.60	0.00
	Chemistry	28	1.14	0.56
	Physics	11	0.84	0.41
Total Scale	Biology	31	1.51	0.32	2 67	0.53	0.58
	Chemistry	28	1.49	0.34
	Physics	11	1.43	0.23

Source(s): Authors work. Combined data from BSc and TQD programs

Table 8

Differences between BSc and TQD preservice science teachers’ abilities related to engineering design processes

Scale	Program	N	M	SD	df	F	p
Identify the problem	TQD	39	1.93	0.68	1	3.77	0.05*
	BSc	31	2.14	0.56	68
Investigating solutions	TQD	39	1.50	0.53	1	2.81	0.10
	BSc	31	1.35	0.47	68
Possible Solutions	TQD	39	1.83	0.33	1	7.93	0.01**
	BSc	31	1.66	0.41	68
Planning	TQD	39	1.64	0.70	1	0.00	0.96
	BSc	31	1.65	0.65	68
Production	TQD	39	1.03	0.35	1	0.97	0.33
	BSc	31	1.09	0.42	68
Testing	TQD	39	1.89	0.73	1	0.25	0.61
	BSc	31	1.95	0.86	68
Communicate	TQD	39	0.99	0.30	1	3.36	0.07
	BSc	31	0.87	0.43	68
Improvement	TQD	39	1.19	0.56	1	0.72	0.39
	BSc	31	1.11	0.53	68
Total Scale	TQD	39	1.50	0.32	1	0.15	0.70
	BSc	31	1.48	0.32	68

Note(s): ^*Significant at 0.05 level; ^**Significant at 0.01 level

Source(s): Authors work

Disclosure statement: No potential conflict of interest is reported by the authors.