Many countries are looking for ways to enable students to engage more effectively with PhD study. This paper aims to consider the effects of explicit discipline-specific research skill development embedded in multiple semesters of an undergraduate degree on PhD preparedness.
This case study of one Bachelor of Health Science programme determined the effectiveness of the implementation of a conceptual model, the Researcher Skill Development framework, across the undergraduate degree programme. Data were gathered through interviews of 9 academic staff and 14 students in their fourth year of undergraduate study, which is a research-focused year.
All students and academics stated the benefits of the use of the Researcher Skill Development framework in undergraduate study including: deepening metacognition of research processes; assisting students toward acting and thinking like researchers; and the research-capacity building of the school. While all academics and all but one student recommended that the framework be used early in the degree programme, a number of interviewees specified problems with the existing implementation of the framework.
While the results are not generalisable, the approach is worth studying in other degree programme-wide contexts to determine its broader capacity to enable students to be more research ready for PhD study when compared to current practice.
When adapted to the context, whole-of-degree research skill development may enable developing countries to have more students and developed countries to better prepared students commencing PhD studies.
No studies currently provide results for explicit research skill development across a degree programme, or of the benefits of this approach for PhD preparation.
Willison, J. and Buisman-Pijlman, F. (2016), "PhD prepared: research skill development across the undergraduate years", International Journal for Researcher Development, Vol. 7 No. 1, pp. 63-83. https://doi.org/10.1108/IJRD-07-2015-0018Download as .RIS
Emerald Group Publishing Limited
Copyright © 2016, John Willison and Femke Buisman-Pijlman Published by Emerald Group Publishing Limited.
This article is published under the Creative Commons Attribution (CC BY 3.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial & non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/3.0/legalcode
Increasing the proportion of students commencing and completing PhD study is a priority in nations with developing higher education systems such as Cambodia (Om, 2011), China (OECD, 2008) and Fiji (University of the South Pacific, 2013) or with large changes in university student demography, such as in South Africa (Mafenya, 2014). In countries with established higher education systems, such as the USA, England and Australia, there is some controversy about whether there is a need for increased completion numbers or rather for students who are better prepared when commencing PhD and who complete it with a deeper and broader skill set (DIISR, 2011). In all cases, strategies to nurture research skills well in advance of PhD are being sought and implemented in a range of contexts. Strategies to improve PhD preparedness come under the umbrella of undergraduate research and range from resource-intensive mentored summer scholarships to approaches that are based in the curriculum (Jenkins and Healey, 2009). Studies have asserted the benefits of mentored undergraduate research experiences to increase PhD preparation (Kardash, 2000; Lopatto, 2004); however, these have only been found to confirm a research-orientation in students already interested in postgraduate research; they do not seem to motivate students who were not considering this previously to participate (Hunter et al., 2007). Moreover, mentored internships do not reach all students, for example, those who may have latent capacities for postgraduate research. A few single-context studies have shown useful, but possibly short-lived, positive outcomes from the development of research skills in semester-length courses in the regular curriculum (Chaplin, 2003; Hoskins et al., 2007; Luckie et al., 2004).
One promising approach to nurture research skills of all students in undergraduate degrees is the use of a conceptual model called the Research Skill Development framework (Willison and O’Regan, 2006/2015) in curriculum and assessment design. Nine years after the first version of the Research Skill Development framework was published (Willison and O’Regan, 2006/2015, 2007), it is being piloted at the institution level in Australia (Monash University, 2015), the USA (University of Wisconsin Stout, 2015) and in nations of the Pacific (University of South Pacific, 2013), as well as informing individual discipline approaches in countries as diverse as Cambodia (Om, 2011) and England (Burkill, 2009).
Evaluation of use of the Research Skill Development framework
The Research Skill Development (RSD) framework has been used and evaluated in a number of individual semester-length course contexts in disciplines as different as Biomedical Sciences (Munns and Chilton, 2014), Business (Willison et al., 2009a, 2009b), Engineering (Cochrane et al., 2009), English (Osborn, 2012), Nursing (Pretorius et al., 2013) and Physics (Menke, 2013), as well as interdisciplinary studies (Venning and Buisman-Pijlman, 2013). A multi-institution study showed that use of the RSD framework could effectively help individual educators and small teams to design semester-length courses that developed students’ discipline-specific research skills in many disciplines and year levels (Willison, 2012). However, the study also showed that benefits of explicit research skill development could be lost unless there was follow-on explicit development. This paper presents the first study showing the impact of explicit and coherent research skill development and assessment embedded in the curriculum over the time span of an undergraduate degree programme (Willison, 2012). Such studies are required to determine the long-term efficacy of upskilling and motivating undergraduate students toward PhD studies (Salter and Atkins, 2014). A model that embeds research skill development in the curriculum is of particular interest because it has the potential to be more efficient and more equitable and has a broader reach than mentored undergraduate research models; there is also evidence that some research skills, such as posing researchable questions, may be better developed in-curriculum than in mentored models (Willison, 2012).
The research presented here focused on the effectiveness of in-curriculum use of the RSD during the undergraduate degree in preparing students for PhD studies. Students were enrolled in a research-orientated, non-compulsory, fourth and final year of a Bachelor of Health Science, called the Honours year. Many students were exposed to the RSD during their undergraduate degree from the first year on. The Honours year embedded an extended version of the conceptual model, called the Researcher Skill Development (RSD7) framework (Willison and O’Regan, 2008/2015). Next, the RSD7 is described, followed by the research context, methodology, results, discussion and conclusions.
Conceptual framework: the Researcher Skill Development framework
The RSD7 framework is based on the RSD (Willison and O’Regan, 2006/2015), with the only difference being an extension of the original five-level continuum to seven levels of autonomy, and so it is called the RSD7 for short. The framework was extended to seven levels to bring in the unequivocally “capital R” research, so that the whole university community would be on the same continuum, from a first-year student to a high-profile professor.
An immediate benefit was the uptake of the RSD7 by academics involved in the Bachelor of Health Science to guide the assessment during the Honours years. The structurally similar five-level version of the framework, had not been taken up at Honours despite its use for six years in the Bachelor of Health Science with first-year students.
The RSD7 is a conceptual framework that is deliberately general in nature, so academics can adapt it to their context. It is a conceptual framework rather than, say, a rubric because it can be operationalised in many ways. The advantage of providing a more generic structure that can be adapted into more explicit rubrics was used in the VALUE rubric which was validated in a large range of degrees and institutions. This rubric was developed by the Association of American Colleges and Universities (Rhodes and Finley, 2013), and its generic nature helped with the uptake in a large range of settings. The VALUE rubric does not provide a range of descriptors for the levels of student autonomy; however, the RSD7 was salient for the research-oriented year of this study due to its articulation of researcher autonomy.
The RSD7 framework has been used in numerous ways, such as facilitating conversations between PhD students and their supervisors (Velautham and Picard, 2009); conversations with indigenous students about empowerment and knowledge; introducing students to necessary aspects of research processes; and for analysing teaching, learning and assessments elements in curricula. In this study, the prevalent mode of use was to scaffold the creation of marking rubrics.
This study evaluated the use of the RSD and the RSD7 to frame research skill development and assessment across a degree programme to prepare students for PhD study.
Facets of research
The RSD7 (Figure 1) describes the processes associated with research in either disciplinary or interdisciplinary contexts as “facets” of research. The RSD7 facets describe what researchers do. These facets have underlying commonalities across disciplines, as oppose to research products that may vary markedly in structure and purpose. The facets are not generic skills, as this would imply a ready transferability, but rather they are overarching perspectives about the research processes that are common across disciplines. In use, these general descriptions are made real by academics who operationalise them as discipline-specific and context-sensitive descriptors (www.rsd.edu.au). The six facets are that researchers:
Embark and clarify: Researchers embark on research, determining and clarifying the need for knowledge while taking note of ethical, cultural and social/team considerations.
Find and generate: Researchers find information and generate data relevant to the research using appropriate methodologies.
Evaluate and reflect: Researchers evaluate information and data and reflect on all processes used.
Organise and manage: Researchers organise information and data, and manage the research process of individuals and of teams.
Analyse and synthesise: Researchers analyse information and data, and synthesise new knowledge to produce coherent individual and team understandings.
Communicate and apply: Researchers write, speak and perform the processes, understandings and applications of the research, and respond to feedback, mindful of ethical, cultural and social/team issues (based on Willison and O’Regan, 2006/2015).
The separation of research processes into six distinct facets is, of course, artificial, as these facets frequently co-occur and overlap. Moreover, research is often characterised by a lack of clarity, fuzziness and even messiness, and therefore, the process is certainly not linear. It may be that the neater, more coherent and linear final phase is most commonly reported in journal articles, but this hides the actual processes used in research into the unknown. However, delineating facets associated with research makes the processes involved explicit, and for many students this equates to being learnable. Delineation is a first step towards demystifying research.
Seven levels of student autonomy
A second step in demystification afforded by the RSD7 is to clarify how much scope students have in initiating research projects and determining processes, and in negotiating final solutions and communications. The specific task or assignment sets the boundaries of autonomy. Making student and supervisor expectations of scope and autonomy clear may be a major factor in a successful mentor-mentee relationship, and one for which the RSD7 has provided effective guidance (Velautham and Picard, 2009). A common anecdotal complaint from students in the Bachelor of Health Science was that they were initially only given very limited amounts of autonomy in undergraduate courses, while they were expected to show independent research skills in Honours. This was also evident in one (Drew et al., 2002) but not in another (Allan, 2012) study of Honours students.
In the RSD7, the amount of scope is called “Extent of Researcher Autonomy” and is represented as a continuum delineated by seven levels. Considering the end of the continuum with minimal autonomy, Prescribed Research (Level 1 of Figure 1), the supervisor models discipline-specific approaches of how to commence, including question framing, hypothesis posing and/or goal setting. The student-researcher follows highly prescriptive guidance, such as tightly defined procedures or guided readings.
Bounded Research (Level 2) requires some limited student-researcher decisions, for example a choice between two given methods. The shift from Prescribed Research to Bounded Research is not trivial, as some beginning researchers struggle when decisions are necessary or grey areas are to be considered.
With Scaffolded Research (Level 3), the parameters of the research are determined by the supervisor, but student-researchers will need to make numerous decisions about what do and so work with a high degree of independence within the parameters set.
The move to Level 4 is into the zone of “researcher instigated” and so has corresponding difficulties: Self-initiated Research describes a move that is instigated by the researcher, with support and advice provided by the supervisor. Researchers may, for example, identify gaps in the literature as the launching pad, select appropriate methodologies and apply analytical techniques, and the supervisor ensures that directions are attainable within the parameters of resources and time.
In Open Research (Level 5), the research is initiated and propelled by the researcher in his or her own direction using self-chosen methodologies and audiences. At this level of autonomy, supervisors provide advice rather than prescriptions. Parameters still exist in terms of disciplinary conventions, and so Open Research does not allow for lack of rigour, as the degree of rigour to be applied is appropriate to the year-level of study and disciplinary norms.
The move to Level 6 is into the realm of discipline leading. The distinctive feature of the move to Adopted Research is where directions chosen and outcomes of research are used by others in the field, for example, by citing, adopting methodologies, being informed by research outcomes or following gaps the researcher has identified.
Enlarging Research (Level 7) is where the researcher changes the nature of the conversation in the discipline, e.g. through a reconceptualisation of what is worth researching or in the development of major research methodologies. This is research that reshapes or enlarges the parameters of the discipline, or conceivably consolidates and refocuses its mandate. Individual researchers may participate in research teams that make a broad contribution that is widely cited or used, even when they themselves have their directions prescribed, bounded or scaffolded by supervisors. In a team then, different personnel may be operating at different levels of autonomy, depending on their roles. Here, individual contributions combine into something bigger than the sum of the parts because the research-leader is operating at a higher level, moving into areas that the field follows.
The six facets of research form the vertical axis of the RSD7 framework, and the seven levels of autonomy comprise the horizontal axis. The resulting table cells are populated with details appropriate to these facets for each specified level. The elaboration of the original framework to include Levels 6 and 7 and so bring in the “discipline leading” aspects of research was introduced in 2008 (Willison and O’Regan, 2008). The RSD7 has been used early in PhD candidature (Velautham and Picard, 2009), for doctoral education more generally (Matas, 2012) and masters level (Willison et al., 2009a, 2009b; Venning and Buisman-Pijlman, 2013), but not previously been considered in an undergraduate research context.
The research took place at School of Medical Science, which is part of a faculty of Health Sciences in one of the research-intensive universities in Australia. In this context, a “school” is an administrative umbrella for several cognate disciplines. The School teaches into the Bachelor of Health Sciences, Bachelor of Sciences, MBBS, Bachelor of Dentistry and Bachelor of Nursing and offers several postgraduate degrees. The School has a strong focus on Honours which, as noted, is the standard pathway to PhD study at the moment. In total, 39 academics and around 80 title-holders and affiliate staff, 35 Honours students and 57 PhD students were associated with the school at the time of conducting interviews.
The School had two early adopters who consistently used the five-level version of the RSD framework from 2005 with large first-year courses in which student numbers increased from 90 in 2005 to 400 in 2013. Uptake of the framework from 2005 occurred on the initiative of individual course coordinators in undergraduate courses. The second author had been working with the RSD7 from 2008 in postgraduate by coursework courses (Venning and Buisman-Pijlman, 2013) and towards the end of 2011, initiated a conversation about the integration of the RSD7 through all the years of the degrees. The School quickly moved to use the RSD7 to conceptualise how to frame Honours-Year assessment in two of the three disciplines of the School commencing in February 2012 (with the third discipline following in 2013), and the resulting marking rubrics that were framed by the RSD7 are publicly available (Medical Science Honours Marking Rubrics, 2012). The way the RSD7 was operationalised was to provide students with task-specific assessment criteria in matrices (rubrics) that are structured according to the six facets; these provide the grade descriptors for tasks, clarifying the expectations for an assignment at a certain year level and autonomy level (Medical Science, 2012). Rubrics in the USA are typically used for quality control of the curriculum design at the university level (Kuh et al., 2014), but RSD-inspired rubrics were directly used at the School of Medical Science in this study with the intention of improving students’ research-oriented learning, including through the tools of assessment and feedback.
A qualitative methodology was used to gain fine-grained data to determine the benefits and detriments of the implementation of the RSD7 in preparing students to be research-ready for PhD study, with the following specific research questions:
What are the benefits and disadvantages of the use of the RSD7-framed rubrics in the Honours year from academics’ and students’ perspectives?
What do Honours students and academics recommend about RSD/RSD7 use in the entire undergraduate degree programme?
Ethics approval for the study was gained from the University’s Human Research Ethics Committee, as part of a the larger study spanning different faculties, which was funded by an Australian Government grant from the Office of Learning and Teaching. Interviews were arranged and conducted in keeping with the ethics protocol in May and June 2012. After informed consent of participants was obtained in writing, the interviews of approximately 1 h duration were conducted face to face and audio recorded. A semi-structured interview protocol (Wengraf, 2001) with 14 questions was used, having been piloted in other contexts (Willison, 2012) and fine-tuned for this context. Criteria for reporting qualitative studies (Tong et al., 2007) were addressed in developing the methodology. Purposive sampling was used to gather interviewees with the intention to collect information from students and staff with a range of exposures and experiences of the RSD7. The second author and another academic in the School both suggested a broad range of students in terms of research competence and potential experience in Honours; they also recommended staff for interviews based on diversity of background, year level each taught, attitude to teaching innovations and familiarity with the RSD7, ranging from no awareness, to extensive use of the framework with students.
Of the 35 Honours students in the School in 2012, 14 were invited to be interviewed and all attended the interview (ten female, four male). This represents 40 per cent of the Honours students and was representative of the gender balance of the Honours student cohort. Four students opted to be interviewed in pairs, in keeping with ethics requirements that students could choose to have a companion in the interview; this resulted in a total of 12 interviews with students; however, each student response was distinguished in the transcription process to keep their responses separate. The Honours students were selected from the two disciplines that were using the RSD7-framed rubrics in Honours in 2012. Of the 39 academic staff in the School, 13 (33 per cent) were invited to be interviewed and nine (six male, three female) attended the interview individually (69 per cent accepted), comprising 23 per cent of academics in the School and representing the gender mix of staff.
Data collection and analysis
During the semi-structured interviews, additional probing took place when an answer seemed incomplete, in conflict with previous answers or worth seeking additional information. The 21 interviews in total yielded a range of benefits and disadvantages of the School’s use of the RSD7 in Honours, and when no additional information emerged, suggesting near-saturation of data, no further interviews were arranged.
The top-level extraction of data included all student or academic comments that addressed the research questions: benefits and disadvantages of RSD7-framed rubrics in Honours and recommendations for RSD7 use in years one to three of Medical Science degree programmes. The first author identified emerging themes which broadly conveyed student and academic sentiments of the benefits. The second author fine-tuned the initial phrasing of categories so that they reflected more accurately the data, and this took place in an iterative process until there was agreement on coding between both authors. Further commentary on the internal consistencies of themes in the emerging paper was provided by four independent peers, which led to clearer phrasing of each theme. A presentation of data and preliminary analysis was provided to a School meeting attended by 35 academic, professional staff and students, and attendees were invited to give feedback on the analysis of the findings, a process which provided confirmation of emerging themes.
Of the 14 Honours students interviewed, 10 had experienced RSD-framed rubrics in two consecutive first-year Human Biology courses, while four students had not encountered the use of RSD-framed rubrics previous to the Honours Year. All Honours students expressed knowledge of the RSD7-based marking rubrics for their assignments and projects during Honours, and one indicated seeing the RSD7 framework itself. Of the nine academics interviewed, six had direct exposure to the RSD7 framework previously and three were exposed to it during the interview for the first time. The RSD7 framework was implemented through multiple task-specific rubrics, framed by the six facets of research and used throughout the Honours year, including with a literature review, a short research proposal poster presentation day and a final research paper (Medical Science 2012): “So we are using it consistently for all of the assessment items” (Academic 3).
The results below are structured to address the research questions. Italics in quotes from the interviews are added for emphasis.
Benefits of the RSD7 implementation in Honours year
There were two major themes concerning benefits of the School’s use of the RSD7-based criteria.
Student metacognition lifted capacity as a researcher.
All 14 of the Honours students indicated that the RSD7-informed marking criteria aided their comprehension about research processes, as the criteria were:
[…] a good guide because you can see all the levels; you can see where you are. You compare yourself to the data. It takes a skill to be honest to yourself; that’s the first skill. When you look at the different levels, you can see where you are fitting and then you look at the levels ahead, at what are your areas of improvement so you can improve yourself […] (Student 1).
This use of “see where you are” and “look at the levels ahead” gives the idea of a conceptual map for discipline-specific processes involving a self-revealing of each student’s position and where he or she needed to go. This clarity of understanding through making the required skills explicit enabled students to also be honest with yourself, and revisitation of these skills on multiple occasions enabled one to improve yourself. Self-honesty and self-improvement are hallmarks of metacognitive processes, where students are aware of their own cognition and actively strive to improve.
An important part of improvement through the RSD7-based criteria was gaining early insights into the expectations of those who would be assessing the complexities of the student research where:
[…] you know exactly what’s required of you, so you’re aware of it in everything you do […]. I think it’s better to be clear from the beginning that that’s what’s expected of you (Student 9).
For Honours, six different rubrics used throughout the year were all framed by the facets of the RSD7, revisiting and so reinforcing the explicated research skill set in discipline-nuanced language. For Student 9, being able to see the overall picture from the beginning was critical for enabling whole-year, increasingly metacognitive application, rather than being uncertain in the early months of Honours study. Moreover, students perceived that the clear guidance provided by the RSD-framed rubrics, even when compared to other rubrics students had experienced. One student, whose enrolment had been mid-way through 2011, provided a comparison between rubrics which were not framed by the RSD7 and those that were organised according to the RSD7, saying:
It would have been good to have it last year […] it’s more specific in terms of the structures and it’s easier to see where you need to improve your strengths or weaknesses (Student 3).
Interviewed students typically found that specific criteria given to them in advance made the research process explicit and visible. However, written criteria on its own did not guarantee that students could independently interpret the RSD-based rubrics:
[…] in my first assessment, I was assessed to be in the 2B range, and I was of course not happy about that, which is why I spoke to my coordinator, and he explained it, because we didn’t know if you are following your recipe, like, the supervisor’s recipe, or is this your idea […]. So that to me all okay, I need to make it very clear in my writing, and from now on in my theses, that this is my idea (Student 14).
For this student, the criteria were not self-evident, but rather required a clarifying conversation, which helped the student realise that he needed to delineate what his own work was, so that assessors would know. The RSD7-based rubric criteria merely articulated this issue in writing, but the conversation with his supervisor clarified the student’s needed response.
One supervisor of Honours expressed his appreciation of how difficult it could be for students to understand the RSD7-based rubrics early on, and suggested social and cognitive elements including that:
[…] you need wine and pizza when you introduce this […]. get them to mark somebody else’s work (Academic 8).
Whether understood through their own reading, or through conversations, the RSD7-based rubrics consistently led the students to see beyond the products of research and their standard structures, to the processes underlying research:
Because I was exposed to research experience, hands-on research experience, in terms of being in a lab and working in a lab and understanding that, okay, it’s not just about having an abstract, an introduction and discussion; it’s about how all of that work that goes into it and it actually comes together at the end. Sort of understanding the broader scope of things made more sense for me, personally (Student 10).
All nine academics indicated that the RSD7 framework fit the research processes of Medical Sciences. One academic, for example, stated:
I think a lot of people might do this sort of thing intuitively, and what this [RSD7] has done is just spelt out what probably the good teachers were trying to do in any case […] (Academic 8).
One of the three academics who had not seen the RSD7 previous to the interview provided a nuanced understanding of the guidance that could be provided by the RSD:
I see the framework being what I’ll call nebulous enough for everybody to be able to accommodate it. Because it’s not very prescriptive. It just sets a framework, and everybody can work within a framework […] [yet] it’s relatively comprehensive in what it describes over all (Academic 2).
The oxymoronic quality of “sufficiently nebulous yet relatively comprehensive” is a key quality of the RSD7 framework, where student researchers and those supervising them should not feel overly constrained and yet do need to have a sense of guidance and a standard for comparability. A student expressed the same sentiment:
So I think it does help that it’s not completely structured, but at the same time that structure also really, really helps (Student 5).
Six students indicated that it was the emerging awareness of the nature of the whole research enterprise together with their own metacognition of the skills required that assisted them towards acting and thinking like researchers. One student stated:
I suppose being exposed to that framework in the Honours year as well, I sort of felt […] when you’re out there on your own as a researcher, that’s essentially what you do (Student 10).
School’s research capacity and profile.
The second major theme concerning benefit for the use of the RSD7 in Honours was around building the School’s research capacity and profile, noted by five academics:
[T]here is an increasing pool of people who are maybe becoming very enthusiastic about how this will get them better students or raise the profile(Academic 3).
These benefits of the explicit use of the RSD7 to improve undergraduate student preparation through metacognition may have application to an identified need in the school concerning further research study:
In working with PhD students, the thing that often leads to conflict is a conflict of expectations and a mismatch of those expectations. Something like this [RSD7 framework] lets students know where a PhD supervisor expects them to be and where a PhD student thinks they should be (Academic 4).
The RSD7 was also perceived as a way of drawing in and interesting research-focussed staff in undergraduate teaching and funnelling their energies:
I think the staff who do a lot of research who are taking some of these courses [that use the RSD7], or running some of these courses, I think they probably see the link more than some of the staff who mainly just teach […] [and never] talk about the link between that and research (Academic 9).
From this perspective, those who are primarily researchers made the link explicit between teaching and research by using courses with the RSD7 embedded to build the research capacity of students. As part of a school with a large number of research active staff at various stages in their career, some academics saw that the RSD7 was a useful part of research capacity building, not just of the students, but also of the academic staff themselves:
If you’re developing the skills of researchers or colleagues or whatever, there needs to be some sort of feedback that, if you like, gives some idea of where the strengths are and where the weaknesses are, if these people are going to be writing research grants and whatever […] (Academic 3).
As an example of this capacity building, one academic applied the RSD7 to herself:
I was getting a bit scared when I was looking at level 7. Gosh, am I working at this level? Which is where you would expect academics to be working […]. I think it could be useful for promotions, when you’re looking at – particularly when you’re going from level C to level D […] this is really what they’re marking: where are you sitting in the international arena? (Academic 5).
This academic saw the potential of the RSD7 to inform her and others’ research trajectories, ultimately enhancing the research profile of the School.
Disadvantages of RSD7 implementation
The major theme emerging as a disadvantage, stated by 8 of the 14 Honours students and 4 of the 9 academics, concerned problems associated with rubrics based on the RSD7, and their use.
The most common statement from students was about specific assessment criteria:
I kind of find that sometimes they [School-wide RSD7-framed rubric criteria] don’t apply very well to the assignment […] our projects are vastly different […]. I don’t understand how the same marking scheme can apply when it’s so specific for what it’s looking for […] one of the ones for the lit review was use of up-to-date relevant literature. It’s like, I don’t have any […] (Student 11).
“Up-to-date literature” was a criteria not perceived to be relevant, as the pertinent literature base on the topic was from 30 years previously according to the student. More worrisome for this student was that the assessors might not appreciate the context of her project, and so would not know the current state of play in the relevant literature. One criterion that two students perceived to be impossible for markers to make valid and informed judgements about “for outstanding grade” was:
Objectives clear, focussed and innovative, extending past supervisor guidelines (Medical science, 2012).
One student remarked about this criterion:
Some of it, you think, how will assessors know that we’re doing that? […] Assessors] don’t know what we and our supervisors talk about in our meetings, so they don’t know whether or not we’re going beyond our supervisors. I think that’s a bit stupid at times (Student 5).
From the perspective of the student, this was a criterion that required mind reading on the part of examiners. As another reason for the lack of applicability of the RSD7-generated criteria, one student stated:
I’ve found out that assessors are also quite subjective. I have one assessor that says I am in level 4 and 5; I have another one that says I’m in level 2 and 3 […]. So I’ve found that this framework is really subjective […] although they make markers a lot more accountable (Student 4).
This student is understandably troubled by the perennial problem of inter-rater reliability, and there is no surprise that mere criteria do not guarantee reliable scores, as this is frequently reported in the literature (Sadler, 2009, 2014). While the student was perplexed by the need to deal with multiple subjective perspectives at once, she appreciated that the markers would at least be able to defend their decisions for where they allocated marks, even if they did not agree with other markers.
Two students noted another problem:
We get a rubric for every single assessment, and they’re all completely different. They’re very much for each assessment, and I hate them (Student 11).
These comments suggest that some students were not aware of the common framing of the RSD four months into Honours, seeing each rubric as separate “for each assessment”.
Two academics noted an issue affecting applicability based around their use of the RSD-based rubrics, especially in oral presentations:
[…] what looks really clear on a piece of paper in an assessment task list becomes a lot more complex. Say you’re in a seminar and someone is talking about a particular area and trying to tease out a quite complicated presentation so that you can tick off numerous little boxes can be quite difficult (Academic 6).
For some academics, assessing with complex criteria is clearly a barrier, especially in oral presentation setting where details like small font size could make the process untenable. Another disadvantage of the use of the RSD-framed rubrics, even though identified by only one academic, may nevertheless be important:
There’s also some attempt to explicitly link assessment items with the Research Skill Development Framework, but there’s also the issue, you’re trying to micromanage how people mark […] (Academic 6).
This touches on tensions around academic autonomy versus a shared conceptualisation, and this tension will be revisited in the discussion.
Recommendations for undergraduate from first year to third year
Notwithstanding the above problems identified, all but one Honours students, and all the academics stated that the RSD7 could be used in beneficial ways to frame undergraduate study before Honours. One student was able to reflect on lessons learned during explicit development of his research skills in two consecutive first-year human biology courses, and clarified how they benefited his Honours study three years later:
It [the RSD rubric] tells you a lot, because I remember one of the criteria was to be specific […] in my [First Year] Human Biology course […] so I think it’s very relevant to Honours, because when we were doing our first seminar, some of the topics were really broad. Automatically you think: I have to be specific otherwise it’s too broad […] (Student 7).
This is a simple but profound demonstration of the development of research skills, where the student internalised the detail to be specific as well as the carriage of the idea throughout the years of study. Like other student quotes below, and as noted in Section 5.1, he indicated that cognition around research processes had started to become innate, where “automatically you think”. Indeed, a majority of students interviewed (n = 9) asserted that the RSD should be incorporated from first year of the undergraduate programme and continued throughout:
I think that would have been a good thing, especially if it was consistent throughout the years, because by the third year it would be a lot more ingrained and it wouldn’t be a novel thing that you’re looking at (Student 2).
These students perceived a deeper, more implicit internalisation of the research processes through the explicit RSD7 framing in multiple courses in the programme:
Since the beginning [of First Year], they have given us assignments based on this criteria. You might not have liked the assignments, but because they have been consistently applying this structure to all of our assignments, we have come to think that way for science, in the perspective of science and writing […] (Student 4).
For this student, one of the ten who encountered RSD-framed rubrics in the first year of the degree, this previous rich exposure had deep and pervasive influence on his thinking all the way through to Honours year. Here the idea is one of being inculcated into research processes so that these processes become internalised, provoking the sentiment we have come to think that way for science and enabling the process of students becoming metacognitive researchers. Students provided practical advice of how to incorporate the RSD effectively in the earlier years: “The idea is good; it just needs to be, for a better term, dumbed down to first-year level” (Student 11). This student, who was generally against the RSD7-framed rubrics in Honours year, here states the process to develop research skills should be developed in the years previous to Honours, with tasks set at an appropriate degree of rigour and sophistication. Another student considered the need for incremental development over the years:
I always remember these [RSD-framed] marking criteria sheets, but I think if they were emphasised more and you know that they kind of reinforce these facets a little more strongly throughout every year, it would be really, really good, I reckon (Student 6).
In a similar way, an academic phrased incremental development in this way:
But if it’s dribbled through in these undergraduate years, it then makes the Honours a lot easier, and then makes the PhD a lot easier, because a lot of students struggle in their first year [of PhD] (Academic 2).
This “dribbling” also gives a sense of what academics thought was the best ways of administering RSD7: earlier than Honours, but not “injected” forcefully and not over-emphasised. In other words, there was a desire that the RSD7 be used to provide structure and scaffolding, but not to the extent that it was inflicted in ways counter to effective learning or to academic autonomy.
Four Honours students agreed that RSD7 would effectively be used earlier in the degree than Honours, but not in the first year, with two students suggesting second year and two students suggesting third year. One of these student’s hesitation on recommending RSD7 in earlier years concerned not her own experience, for which the RSD7 had clear relevance, but that of her friends:
I loved what I did [in First year], but I just remember a lot of my friends used to sit there complaining, just going, this rubric sheet doesn’t make much sense because why am I ever going to need to do any of this stuff […] (Student 10).
Two students commented that the benefits of the RSD-framed rubrics from first year were negligible, with one saying:
I didn’t pay attention to it [RSD7-framed rubric in First Year], because it was kind of at the back of the assessment. They were never kind of made a big deal, whereas in Honours we kind of get handed this, like, go find this (Student 12).
For this student, it is not in having the rubrics but in how they are used, discussed and emphasised that is educationally pivotal. One out of the 14 Honours Students stated that the use of the RSD was a not a good idea in any of the years before Honours, saying that no a priori framework would be superior because that:
[…] sort of helps us to be a bit more flexible […] [and] sort of makes you think more, I think, because other people have different ideas and expect different things (Student 8).
This student perceived that a pre-determined structure may negatively affect flexible thinking.
While one academic was against an enforced use, eight of the nine academics saw an advantage of the RSD7 to frame an approach in common across all years of undergraduate study including Honours:
So there has to be what I’ll call a program wide approach where all the academic staff need to get together, or as many as possible, so you have the First-Year group there and Second Year and Third Year, and they all learn from one another (Academic 2).
A large majority of students and academics interviewed perceived that the marking criteria framed by the RSD7, located in appropriate courses and assessments across the degree programme, had reinforcing and multiplying effects on learning discipline-specific research skills.
In terms of RQ1, there are clear benefits of the RSD7 for consistently informing learning, teaching and assessment processes throughout the Honours year. These benefits included that the RSD7 framework was perceived to fit closely to the nature of research in Medical Science, enable students to think metacognitively as researchers and help build the research capacity of students, academics and the School as a whole. These points are in agreement with a study of student inquiry that found “when appropriately framed (especially with the relevant student motivation and autonomy), such lessons can become deeply internalized” (Allchin et al., 2014, p. 469, italics added). Disadvantages of the RSD7-framed rubrics mainly focussed on the specific criteria, where some were perceived to be too prescriptive, too vague, subjective or irrelevant, in keeping with research on pre-determined assessment criteria (Adcroft, 2011; Sadler, 2009, 2014).
For both advantages and disadvantages communication was pivotal to the learning enterprise, with effective communication not guaranteed by the use of the RSD7-based marking rubrics, but more likely enabled by conversations framed by them. For example, one student in Section 5.2 perceived that the criterion about going beyond supervisor’s guidelines was “stupid”; however, Section 5.1 presented a student who also referred to this same criteria and found it provided substantial insight after discussing with his supervisor, saying that in reflection “I need to make it very clear […]” The first student said how will assessors know? The second student agreed that assessors wouldn’t know; therefore, I need to make it very clear in my writing. This demonstrates that written criteria alone are insufficient for at least some students to clarify assessment requirements.
Another element exposed above is that the RSD7 prompted academics to make visible criteria that are frequently left implicit for students. The general facet of “embarks and clarifies” was operationalised as the problematic, but necessary, description of “going beyond supervisor’s guidelines”, an honest exposure of assessment of expectations of high-performing students in the initiation of research. However, using the framework to unearth and articulate implicit criteria explicitly does not make the words or ideas immediately sensible or palatable, but preferably would be part of a process, whereby students critically analyse the criteria rather than reactively thinking “stupid”. Based on the differential response between students noted above, the RSD-framed criteria need to be used as a learning and conversation tool at least as much as an assessment tool, so that more students have an it-got-me-thinking reaction. It has been found elsewhere that the RSD framework frequently promotes a shift to making explicit that which has often been left implicit (Willison, 2012).
That some students saw several criteria as irrelevant, such as “recent literature” suggests that these students could better apprehend the standards of the discipline, and take pains to explain why their specific research does not fit the common practice. The criterion should suggest to the student – initially, with guidance from the supervisor, the need to explain why she is relying on what looks like out-of-date material. Overall, the effective use of the RSD7 and its resulting rubrics seemed to require dialogue between students and academics, in keeping with the academic who suggested introducing the rubrics with pizza. It seems that the marking rubrics framed by the RSD7 were like a “frozen conversation”, where the dense articulations of what is being sought needed to be defrosted by human-to-human contact to enable more fluid understandings.
RQ2 was “What do Honours students and academics recommend about RSD7 use in the entire undergraduate degree programme?” All those interviewed, except one Honours student, stated that RSD7 could be used to advantage earlier in the undergraduate degree than Honours. There was a strong emerging sense that across-the-years, explicit research skill development informed by the RSD7 is a way to enable the education needed before and during Honours, and to provide the preparation for PhD study. This may be contrasted with another study which found that without explicit development, there was a risk that research skills may atrophy from one year to the next (Willison, 2012). An explicit and ongoing upskilling process is more likely to overcome the problem that without appropriate structure and guidance, by the end of their degree, students tend to perform open-inquiry projects at “the same level of sophistication as in their introductory core course” (Chaplin, 2003, p. 231).
Repeated exposures of the same framing of the RSD7 maximise the potential for student metacognition, where they make their cognition visible to themselves and, as Student 4 said, because across the years of the degree “they have been consistently applying this structure to all of our assignments, we have come to think that way for science”. Students’ awareness of their own thinking processes are heightened through repeated and diverse exposures to the same six facets of the RSD in multiple contexts, and their potential for self-propelled learning is enhanced as shown by Student 1’s statement “you can improve yourself”. Metacognition was noted to be of paramount importance for fostering professional growth by a study on inquiry skills in science (Michalsky, 2012), and to student growth as researchers in this present study. Student metacognition evident in this study resulted in a dawning appreciation that the specifics of rubric criteria change time and again, but the six facets of the research process are the same, and the boosted metacognition fundamentally improved most interviewed students’ approaches to researching.
The above concurs with a study on students as apprentice researchers (Feldman et al., 2013, p. 240), which found that:
[…] there is need for professors to be more proactive in helping their students gain intellectual proficiency not just as part of doctoral studies but also for undergraduates and master’s degree students (italics added).
The opposite of this are separate courses where students have difficulties seeing the connections in the development of their research skills in different contexts. A lack of conceptual connection may lead to assessment and feedback that does not build systematically, but rather to courses and rubrics that have separate and distinct foci for different products. This may run contrary to reinforced learning across the years:
[…] given the growth of ever more detailed marking schemes for assessments, does feedback become something which is too specific to a single episode of assessment rather than generalisable to the learning experience as a whole (Adcroft, 2011).
A large majority of students interviewed were able to generalise to the learning experience as a whole, where they were metacognitive and considered themselves being prepared as researchers. This reported skill development is also in keeping with other studies of the RSD7 at the postgraduate level, (Venning and Buisman-Pijlman, 2013; Velautham and Picard, 2011) which, together with this study, suggest that the RSD7 could be used in master’s degrees as well as Honours degrees to better prepare students for PhD study. Any programme-level implementation of the RSD would need to balance advantages, such as coherency in the programme, with the dangers associated with over-riding individual academic’s autonomy.
Limitations and biases of the study
This study was conducted with a proportion of academics from one school and students from one honours cohort, so the study cannot be generalised to all the School’s academics and students, or to other schools and honours cohorts. The study risked a conformation bias; however, there is evidence especially in Section 5.2 that disconfirming evidence was gathered and presented, to mitigate this potential bias to some extent. The study was conducted at an Australian university, and this raises the question about generalisation to other settings and counties. The bachelor and Honours year are comparable to, for example, the UK system or the longer undergraduate science degrees where time is reserved for extended periods of research internships. It is noteworthy that a grading rubric based on the RSD7 was used successfully in The International Master of Science in Addiction Studies, which is taught by partner universities in the USA, the UK and Australia (Venning and Buisman-Pijlman, 2013). The consistent use in this current study of a common conceptual framework to inform rubric construction over multiple year levels to inform the students of their progress and to provide guidance on how to improve seems to be innovative. This use of a conceptual framework may require a cultural shift for academics who may otherwise be focussed on collecting the data from rubrics for accreditation purposes (Kuh et al., 2014) without seeing the broader educational issue.
Notwithstanding the above comments, this study is an improvement on studies that have focussed on RSD implementation by several academics (Peirce et al., 2012; Pretorius et al., 2013; Willison et al., 2009a; 2009b) or many academics working individually or in small teams (Willison, 2012): these studies of early adopters risk “early implementer” bias. This current study has considered a whole-school context, where not all those involved in implementation are convinced about the approach. This more closely mirrors the second phase of implementation of an innovation, which, in education, typically is less successful than first-phase pilot studies. Broader studies are needed to reveal more fully the advantages, disadvantages, limitations and successful methods of implementation of the RSD7 at the multiple school and institution levels.
In this current study, the RSD7 provided a conceptual language-in-common which enabled meaningful conversations between academics and students, even if people did not always agree on what was being said. The RSD7 framework was perceived by academics and students to fit closely to the nature of research in Medical Science, and by operationalising it as rubrics, academics were prompted to state clearly the criteria that are frequently left implicit. As academics made research skill development explicit over an extended timeframe, students found that research processes were both more visible and were increasingly internalised so that you came to think that way for science. The large majority of those interviewed suggested strongly that multiple and varied exposures to the same six RSD7 facets across the years of undergraduate study enabled substantial long-term benefits, including maturation of their research skills due to heightened metacognitive processes and identification of themselves as researchers. In this way, the RSD7 was perceived to build the research capacity of students, academics and the School as a whole.
The RSD7 opens up possibilities of using a structure which is sufficiently nebulous yet directed for long-term guidance and reinforcement of student research skill development. However, multi-school and whole-of-institution studies are needed to delve more deeply into how effectively the RSD7 may help with the development of student research skills in many contexts, and with the selection of students who are suited for post-graduate research. For countries with sufficient or excess PhDs, as well as for countries striving to increase PhD enrolments, the RSD7 presents the potential for students to be more research-ready on commencement of their research degrees and so have enhanced outcomes in terms of graduate qualities and outputs. In the School of Medical Science in this study, implementation and use of the RSD7 enabled fluid understandings and conversations around research processes and so it was an important part of students becoming researchers and researchers becoming renowned.
Adcroft, A. (2011), “The mythology of feedback”, Higher Education Research and Development, Vol. 30 No. 4, pp. 405-419.
Allan, C. (2011), “Exploring the experience of ten Australian Honours students”, Higher Education Research and Development, Vol. 30 No. 4, pp. 421-433.
Allchin, D., Møller, H. and Keld Nielsen, A. (2014), “Complementary approaches to teaching nature of science: integrating student inquiry, historical cases, and contemporary cases in classroom practice”, Science Education, Vol. 98 No. 3, pp. 461-486.
Chaplin, S. (2003), “Guided development of independent inquiry in an anatomy/physiology laboratory”, Advances in Physiology Education, Vol. 27 No. 4, pp. 230-240.
Cochrane, S.Goh, S. and Ku, H. (2009), “An investigation into the application of research strategies in the final-year undergraduate engineering and surveying projects”, 20th Annual Conference for the Australasian Association for Engineering Education, Engineering the Curriculum, A.C.T., Engineers Australia, Barton, 6-9 December, pp. 193-198.
DIISR (2011), Research Skills for an Innovative Future: A Research Workforce Strategy to Cover the Decade to 2020 and Beyond, Department of Innovation, Industry, Science and Research, Canberra.
Drew, M.E., Subramaniam, N. and Clowes-Doolan, K. (2002), “Students’ experience of the honours’ supervisory relationship: a preliminary investigation”, Discussion Paper No 113, Discussion papers in Economics, Finance and International Competitiveness, Queensland University of Technology, Queensland, available at: http://eprints.qut.edu.au/523 (accessed 1 September 2013).
Feldman, A., Divoll, K.A. and Rogan-Klyve, A. (2013), “Becoming researchers: the participation of undergraduate and graduate students in scientific research groups”, Science Education, Vol. 97 No. 2, pp. 218-243.
Hunter, A.B., Laursen, S.L. and Seymour, E. (2007), “Becoming a scientist: the role of undergraduate research in students’ cognitive, personal and professional development”, Science Education, Vol. 91 No. 1, pp. 36-74.
Jenkins, A. and Healey, M. (2009), Developing Undergraduate Research and Inquiry, Higher Education Academy, New York, NY, available at: www.heacademy.ac.uk/assets/York/documents/resources/publications/DevelopingUndergraduate_Final.pdf (accessed 14 November 2010).
Kardash, C. (2000), “Evaluation of an undergraduate research experience: perceptions of undergraduate interns and their faculty mentors”, Journal of Educational Psychology, Vol. 92 No. 1, pp. 191-201.
Kuh, G.D., Jankowski, N., Ikenberry, S.O. and Kinzie, J. (2014), Knowing What Students Know and Can Do: The Current State of Student Learning Outcomes Assessment in US Colleges and Universities, National Institute for Learning Outcomes Assessment, available at: www.learningoutcomeassessment.org/knowingwhatstudentsknowandcando.html on (accessed 1 October 2015).
Lopatto, D. (2004), “Survey of undergraduate research experiences (SURE): first findings”, Cell Biology Education, Vol. 3 No. 4, pp. 270-277.
Luckie, D., Maleszewski, J., Loznak, S. and Krha, M. (2004), “Infusion of collaborative inquiry throughout a biology curriculum increases student learning: a four-year study of ‘Teams and Streams’”, Advances in Physiology Education, Vol. 28 No. 4, pp. 199-209.
Mafenya, P.N. (2014), “Challenges faced by higher education institutions in research skills development: a South African open and distance learning case study”, Mediterranean Journal of Social Sciences, Vol. 5 No. 4, pp. 436-442.
Matas, C. (2012), “Doctoral education and skills development: an international perspective”, Revista de Docencia Universitaria, Vol. 10 No. 2, pp. 163-191.
Medical Science (2012), “Medical science honours marking rubrics”, available at: www.adelaide.edu.au/rsd/examples/#medicalscience (accessed 1 September 2013).
Menke, C. (2013), “Streamlining program learning objective assessment with targeted activities and descriptive rubrics”, available at: http://assessment.ucmerced.edu/sites/assessment.ucmerced.edu/files/page/documents/menkec-presentation.pptx.pdf (accessed 14 February 2015).
Michalsky, T. (2012), “Shaping self-regulation in science teachers’ professional growth: inquiry Skills”, Science Education, Vol. 96 No. 6, pp. 1106-1133.
Monash Univeristy (2015), “Monash univeristy library: research skills development”, available at: http://monash.edu/library/skills/rsd/ (accessed 21 February 2015).
Munns, S. and Chilton, L. (2014), “Demand evidence and think critically: building research excellence in tomorrow’s scientists”, Proceedings of the Australian Conference on Science and Mathematics Education, University of Sydney, Sydney, 29-30 September, pp. 134-139.
OECD (2008), Higher Education to 2030: Demography, Centre for Educational Research and Innovation, Organisation for Economic Cooperation and Development, Paris.
Om. S. (2011), “Cambodian post-graduate students’ perceptions of their development as beginning researchers”, Thesis submitted for the partial fulfilment of Masters Degree in Education, Royal University of Phnom Penh, Phnom Penh.
Osborn, J. (2012), “The national year of reading: celebrating the role of literature in an academic culture”, Australian Library Journal, Vol. 61 No. 4, pp. 281-288.
Peirce, E., Ricci, M., Lee, I. and Willison, J. (2012), “First year human biology students in the ivory tower”, Proceedings of the Uniserve Science Conference, Sydney.
Pretorius, L., Bailey, C. and Miles, M. (2013), “Constructive alignment and the research skills development framework: using theory to practically align graduate attributes, learning experiences, and assessment tasks in undergraduate midwifery”, International Journal of Teaching and Learning in Higher Education, Vol. 25 No. 3, pp. 378-387.
Rhodes, T. and Finley, A. (2013), Using the VALUE Rubrics for Improvement of Learning and Authentic Assessment, Published by Association of American Colleges and Universities, Washington, DC, available at: www.aacu.org/publications/using-value-rubrics (accessed 3 October 2015).
Sadler, D.R. (2009), “Indeterminacy in the use of preset criteria for assessment and grading”, Assessment and Evaluation in Higher Education, Vol. 34 No. 2, pp. 159-179.
Sadler, D.R. (2014), “The futility of attempting to codify academic achievement standards”, Higher Education, Vol. 67 No. 3, pp. 273-288.
Salter, I.Y. and Atkins, L.J. (2014), “What students say versus what they do regarding scientific inquiry”, Science Education, Vol. 98 No. 1, pp. 1-35.
Tong. A., Sainbury, P. and Graig, J. (2007), “Consolidated criteria for reporting qualitative research (COREQ): a 32-item checklist for interviews and focus groups”, Internal Journal of Quality Health Care, Vol. 19 No. 6, pp. 349-357.
University of the South Pacific (2013), “University of South Pacific Strategic Plan, 2013 -2018”, available at: www.usp.ac.fj/strategicplan2013-2018 (accessed 1 September 2013).
University of Wisconsin Stout (2015), “UW-Stout research skill development cohort”,available at: https://uwstoutrsd.wordpress.com/ (accessed 21 February 2015).
Velautham, L. and Picard, M. (2009), “Collaborating equals: engaging faculties through teaching-led research”, Journal of Academic Language and Learning, Vol. 3 No. 2, pp. 130-141.
Venning, J. and Buisman-Pijlman, F. (2013), “Integrating assessment matrices in feedback loops to promote research skill development in postgraduate research projects”, Assessment & Evaluation in Higher Education, Vol. 38 No. 4, pp. 567-579.
Wengraf, T. (2001), Qualitative Research Interviewing: Biographic Narrative and Semi-Structured Methods, Sage, London.
Willison, J. (2012), “When Academics integrate research skill development in the curriculum”, Higher Education Research and Development, Vol. 31 No. 1, pp. 905-919.
Willison, J. and O’Regan, K. (2006/2015), “Research skill development framework”, available at: www.adelaide.edu.au/rsd (accessed 26 September 2015).
Willison, J. and O’Regan, K. (2007), “Commonly known, commonly not known, totally unknown: a framework for students becoming researchers”, Higher Education Research and Development, Vol. 26 No. 4, pp. 393-410.
Willison, J. and O’Regan, K. (2008/2015), “Researcher skill development framework”, available at: www.adelaide.edu.au/rsd/framework/rsd7 (accessed 26 September 2015).
Willison, J., Pierce, E. and Ricci, M. (2009a), “Towards student autonomy in literature and field research”, paper presented at the Higher Education Research and Development Conference, Darwin.
Willison, J., Schapper, J. and Teo, E. (2009b), “Multiple methods of improvement of research skills in business ethics and business law”, paper presented at the QATLHEBEC Conference, Melbourne.
Bahr, N. and Lloyd, M. (2011), “Course cohesion: an elusive goal for tertiary education”, Journal of Learning Design, Vol. 4 No. 4, pp. 21-36.
The authors thank the colleagues and students of the school that was the focus of this study for their contributions. This study was supported by a grant from the Office of Learning and Teaching, Australia.