Mapping a university's research outputs to the UN Sustainable Development Goals

Abstract

The UN Sustainable Development Goals (SDGs) provide a framework to achieve sustainable development and fulfilling these Goals will take an unprecedented effort by all sectors in society. Many universities and businesses are using the Goals within their strategies and sustainability reporting. However, this is difficult as there is currently no standard methodology to map the 17 goals, 169 targets and 232 indicators. Work at the University of Leicester has focused on developing a robust methodology to map a higher education institution's (HEI's) research contribution to the Goals. We have integrated this unique methodology into an automated software tool to measure a university's academic contribution to the Goals using mathematical text mining techniques. Our ability to quickly and effectively map institutions' research contributions has boosted our ambitions and efforts to develop software to map the full operations of an HEI or business.

Citation

Mistry, A., Sellers, H., Levesley, J. and Lee, S. (2023), "Mapping a university's research outputs to the UN Sustainable Development Goals", Emerald Open Research, Vol. 1 No. 9. https://doi.org/10.1108/EOR-09-2023-0004

Introduction

The United Nations Sustainable Development Goals (SDGs) provide a blueprint to achieve a better and more sustainable future for all by 2030 (United Nations, 2015). In 2015, the UN developed these Goals from the initial eight Millennium Development Goals to make the goals more universal for both the Global North and South. The Goals showcase the wide reach of sustainability including environmental, economic and social sustainability. All the countries of the UN General Assembly signed the 2030 Development Agenda and committed to helping to achieve these Goals. Governments are not only responsible for adopting the Goals, however, as universities and businesses also have a duty to work towards them.

Importance

Universities for good

Education is a public good, allowing students the opportunity to become change makers in the world. Although only around 3% of people go to university globally, 80% of world leaders went to university so education has a disproportionate influence (St George’s House, 2018). Universities play an important part in achieving these goals as “basic and applied research can address real-world problems, societal needs, mind sets, and technologies necessary to break new ground of and for sustainability” (Körfgen et al., 2018) and also in teaching the future generations how to deal with global issues.

Universities are training the world leaders of the future and students believe that their education should involve both critique of the sustainability agenda and information on how to get more practically involved in being part of the solution (National Union of Students, 2019). Universities, with their broad remit around the creation and dissemination of knowledge and their unique position within society, have a critical role to play in the achievement of the Goals. Universities are taking on the role of “change agents for societal transformation at the interface between scientific, political, and societal stakeholders and institutions” (Körfgen et al., 2018). This highlights the importance and need for transformational research not just informational research.

The demand from students is also increasing:

• Approximately 80% of students want their institution to be doing more on sustainable development (National Union of Students, 2019)

• Around 60% of students want to learn more about sustainability (National Union of Students, 2019)

Across the UK higher education sector there has been a change in dialogue with new students in the 2019–20 academic year, as both UK and international students recognise the Goals from their school education and are asking how the University is engaging with them and how they can be involved.

Measuring impact allows universities to fulfil demand for SDG-related education, build new partnerships, access new funding streams, and define a university beyond just statistical results. The Research Excellence Framework (REF) is the current system for assessing the quality of research in UK Higher Education Institutions (HEIs). The REF holds significant importance within HEIs as the outcomes of the assessment inform the allocation of research grants for each institution. It also helps to demonstrate the wider benefits and impacts of the investment into research conducted across the UK. The REF activity is key for HEIs as it enables the assessment of research impact beyond academia. In this context impact is defined as “‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”.

Reporting

The recognition and importance of the Goals are constantly increasing within the higher education sector. In 2019, the Times Higher Education (THE) ran the inaugural University Impact Ranking, which was solely based on the Goals and their indicators, with the number of research publications & citations being a key part in the scoring. Using the Goals as a reporting framework is becoming the norm. Over 90% of the global 250 companies are reporting on them and it is fast becoming a de-facto requirement (KPMG, 2015). But with 17 goals, 169 targets and 232 indicators, it is very complicated to be able to map activities and operations appropriately.

Current problem

With the growing need to rigorously map contributions to the Goals, in line with new reporting and rankings, organisations are looking for ways to map their contributions. However, there is no standardised method to map academic or operational contributions. Other institutions and organisations have started to explore this issue (Körfgen et al., 2018), however there has not been a widely agreed, robust methodology.

Mapping the contributions can be time consuming to gather and audit data from different departments, both operationally and academically. Departments across the institution often work in silos so it can be difficult to gather information and data relating to SDG-related activities across institutions. There is also a knowledge barrier as not everyone has a full understanding of the Goals so collating all the relevant information can take longer. By creating an automated methodology, this could promote breaking these silos and encourage departments to engage with the Goals and easily map their own activities.

Aims & objectives

The objectives for this project are:

• To quantify an institution’s academic contributions, through publications and impact, towards the 17 Goals

• To develop a standardised method to map research contributions

• To use the results from the mapping exercise for new reporting & university league tables

• To create an automated solution to mapping academic contributions to the Goals

This paper aims to describe the development of our SDG mapping tool with a view to contributing to an agreed sector SDG reporting methodology. We also share the results of our investigations and suggest areas of exploration around academic contribution to the Goals that this methodology allows.

Methods

Keywords

We first had to identify a way to categorise research publications into the different Goals. There was no standard list of keywords that represents the 17 goals, 169 targets and 232 indicators so we developed keywords from the Goals’ targets and matched with existing lists of keywords and phrases from the best examples available (Monash University and SDSN Australia/Pacific, 2017). These were refined further to exclude any keywords which produced a high number of outliers. This was done by using a criteria based on the SDG targets where the abstracts of the papers selected were checked against the targets to ensure the context is in line with the specific Goal. We improved the list further using cluster analysis to identify missed words or high frequency words that selected irrelevant research publications.

Mapping research publications

We used the keyword lists to search for academic output - for example journal articles and books. After initially trialling with Excel 2016, we used the Scopus database to search for publications by the SDG-specific keywords. Scopus is an abstract and citation online database of peer-reviewed literature (i.e. journals, books and conference proceedings) from across the world. Query codes were developed for each of the Goals (Mistry & Sellers, 2020a) with the affiliation identification number and the keywords, using examples from the Aurora Network as a baseline (Vanderfeesten & Otten, 2017). There is no formal definition of when research is classed as “research”, so the query code searches for publications in the last five years, in line with the introduction of the Goals in 2015. We started by searching through publications only from the University of Leicester so they could be internally validated. Initially, a manual search using the 17 query codes provided number of publications and citation counts for each Goal. The query codes were refined to reduce the number of outliers that were present by using a criteria similar to the initial keywords where the publication abstracts were checked against the relevant SDG targets.

REF impact case studies

The identification of the REF as significant in the research activities of all UK universities formed the basis for development of the automated mapping methodology. Mapping the REF Impact Case Studies highlights the very best of research activities that take place within a higher education institution and the mapping process allows us to undertake a breakdown analysis of the research submitted and enables us to distinguish which area of the university research has originated from.

For this analysis, REF 2014 datasets were extracted from the REF publicly available website. To ensure data extracted was homogenous, case studies were extracted using the following requirements. The dataset included 34 unit of assessments, which are specific subjects' areas, title and summary of impact section. For each of the 85 case studies submitted by the University of Leicester in 2014, manual analysis of the title and summary of each REF impact case study was undertaken in order to classify the document to one or more relevant Goal. In-depth knowledge of the Goals enabled a broad but detailed picture of the University of Leicester’s contribution to each Goal to be uncovered. The depth of understanding of the Goals provided the basis needed to manually search and analyse the impact case studies. The identification of essential keywords and themes allowed for the classification of the case studies by Goal. This process was repeated for each unit of assessment and helped to understand the institutional breakdown of research submitted to the REF.

The classified data was organised according to number of case studies, per Goal, per unit of assessment (UoA) using Excel. The classified case studies were grouped together according to the University of Leicester college structure. There are three colleges: The College of Life Sciences, the College of Science and Engineering and the College of Social Sciences, Arts & Humanities. Using this information and divisional structure of each college, the case studies were organised within the department or school that they originated from.

Text mining clustering techniques were applied to the organised dataset, using R Studio 1.1.463. Clustering techniques are used for “large-scale topic discovery from text” (Larsen & Aone, 1999). These techniques extracted high frequency word sets from the summary section of the case studies. This process produced basic lists of keywords associated with each Goal. One benefit of using this technique allows for the quick summary of the “contents of a document collection, enabling users to selectively drill deeper to explore specific topics of interest without reading every document” (Larsen & Aone, 1999).

Alongside text mining techniques, a process of manual keyword selection took place which involved manually scanning the ‘summary of impact’ section for each case study within each of the Goals. Using this method, a list of keywords was selected based on our knowledge of the SDG targets. The text mined keyword lists were then combined with the manually selected to create a master keyword list. The initial lists produced did not include keywords for Goals 1 (No Poverty), 2 (Zero Hunger) and 6 (Clean Water and Sanitation) because there are no University of Leicester specific case studies manually classified for these three Goals. As a result, there was a significant limitation of using only the University of Leicester’s data as we failed to produce a full set of basic keywords for each Goal. We believed at this point, it was not effective to use the ‘summary of impact’ section of the case studies as it is very short piece of text and we believed the length of text hindered the discovery of all the words that were relevant to the Goals.

To combat this limitation, the process of text mining and manual selection of keywords was repeated. In this instance, the ‘underpinning research’ section of the REF Impact case studies were used as an alternative. This decision was made as this section of case study text were longer and more detailed, therefore more keywords could be extracted, in turn increasing the likelihood of achieving complete keywords list for each of the 17 Goals. We believed that the words selected from the ‘underpinning research’ text would be more suited to the Goals. Completing this exercise resulted in another set of keywords from both methods of text mining and manual selection for the ‘underpinning research’ section of Leicester’s REF Impact Case Studies.

Although this exercise was useful, it did not generate a comprehensive list of keywords for each of the 17 Goals as there are limitations to using only the University of Leicester’s REF 2014 data. To fill the gaps in the data our next step consisted of repeating the text mining technique using our research publication keywords which covers all 17 Goals. Our use of complete research keywords meant that we were able to obtain better results.

After the process of text mining, clustering techniques were used on the data set. Six main clusters were identified from the text mined keyword lists. The fewer the number of clusters the better and more accurate the results. These word clusters produced 50 high frequency words. These high frequency words were then used to compile a REF impact case study keywords list for Goals 1 (No Poverty), 2 (Zero Hunger) and 6 (Clean Water and Sanitation) which we previously did not have completed lists for. We now have a full compilation of appropriate keywords for each UN Sustainable Development Goal which can be used to classify all UK University REF Impact Case Studies (Mistry, 2020; Mistry & Sellers, 2020b).

Results & analysis

The master list of keywords for the REF Impact Case Studies is the central component of our automated tool. The tool is composed of a PDF reader and a set of coding in which our specific SDG keywords sit. The PDF reader extracts a specific section of text from the case study document which is then examined by the coding and classifies the case study to the relevant Goals. In many cases, more than one Goal can be classified within each document. The automation of the classification process sets our methodology apart from manual classification as it removes the inevitable degree of human error that can occur when manually classifying documents no matter the level of knowledge about the Goals.

It is important to note that due to the difference in text format, length and style, the REF impact case study keywords are different those previously compiled for research publication mapping. This demonstrates that each function of the tool (research, impact, teaching & operations) requires a different set of keywords. Acknowledging this means that we are able to select the most suitable set of keywords dependant on what function of an organisation is being mapped.

The methodology we have produced provides a high-level picture of a HEI’s contribution to the Goals, in terms of research publications and impact. The manual method used to acquire Goal specific research publications is time consuming however, due to the modular nature of the coding we are able to replace the impact keywords with the research keywords in order to automate the retrieval of research publication data. The flexibility in our methodology is a major achievement within our project.

The flexibility of the methodology is once again being used to develop to complete more detailed analyses of an institution, identifying where research and research impacts originate from within an institution. Looking at Leicester’s research at a more micro level, we can undertake mapping exercises according to the University’s college structure to determine how our college specific research aligns with the Goals, enabling us to demonstrate our research strengths and weaknesses. Furthermore, further divisions within colleges can also be used to create analysis of impact of research. This is important as schools and departments inform University research strategy and direction. The more focussed the mapping exercise the more we can learn about our research patterns in line with the Goals.

For example, the results show that University of Leicester contributes to every Goal between the years of 2014 and 2019 (Figure 1) but also highlights Leicester’s strengths and weaknesses in relation to the Goals. The three highest goals (Goals 3, 16 and 10) are mostly related to social sustainability, followed by environmental sustainability (Goals 15, 11 and 13). The Goals with the lowest contribution are Goals 14 and 4.

By looking at both the number of publications and their citation counts, we can gain insight into the impact and reach of the publications. When comparing the number of publications to the numbers of overall citations from these publications, there are some similarities and surprising differences (Table 1). The top two goals with the highest publication numbers also had the highest amount of citations.

When looking at the differences, Goal 6 had 118 publications but had a large amount of citations (10,823) which was surprising (Table 2). Other goals with a high citation count with lower publication numbers include Goal 5 (155 publications; 7,318 citations) and Goal 2 (157 publications; 6,393 citations). This shows that even though the output was relatively small, it had a large amount of impact in terms of citations. Goal 13 (Climate Action) also moved up one place in the ranking from 6^th highest number of publications to 5^th in the citation count.

When looking at University of Leicester’s REF Impact case studies from 2014, we found that Goal 3 (Good Health and Wellbeing) had the highest count of case studies, similarly to the findings when looking at publications (Figure 2). However, Goal 8 (Decent Work and Economic Growth) and 4 (Quality Education) also had a high number of case studies, despite being low in the publication mapping. Whereas other goals that had a high number of publication numbers and citation counts, only have a couple of REF case studies, for example Goals 16 and 10. This shows that different mapping is required to see the whole contribution towards the Goals and capture both output and impact.

Visualisation

The visualisation of the SDG mapping data was difficult to make the data user-friendly and engaging, as there were 17 data points in a wide range. Bar charts and line charts had to be extended to contain the 17 Goals, which meant that the graph was hard to read. Pie charts were also hard to read due to having 17 segments with varying sizes. We adapted a version of the filled radar chart to give each data point a varying height while remaining in a circular position to ensure all points can be seen. The icons were added separately to provide an indicator for the relevant Goal for each section and the section colours match the official SDG colours to keep in theme.

Implications

Choosing the right keywords to use in the query codes was difficult to ensure only relevant publications or case studies were found. One implication was that many of the keywords have multiple meanings; for example, Goal 11 (Sustainable Cities and Communities) involves looking at transport systems, such as public transport, however transport can also be used in other contexts like transporting molecules around the body. Therefore, it was important to identify these double meanings and use two worded keywords or phrases to ensure it was identified in the right context.

There were also implications with using Scopus as the results are assuming that there are no duplications of publications in the database. Even though Scopus is an extensive database, there is a possibility that it might not hold all the publications from the University so some might be missed in the results.

With this methodology, we are only capturing published research and does not capture active research that is going on within institutions. This limits how we can map the current contribution towards the Goals as publication is at the end of the research process. There is a possibility to use this mapping methodology earlier in the process too, such as at the funding stage to capture research at different stages and see how it changes throughout the process.

Discussion/Conclusion

This methodology has many applications by providing data to inform and advance positive impact. The methodology can help to measure progress towards the Goals through mapping the impact of research, which can be done on a regional, national or international level depending on the research. In particular, the REF Impact case studies can provide information about impact on communities and government on varying levels.

HEIs have a role in achieving the Goals so it is important that they are used to inform decision-making and strategies (Ruiz-Mallén & Heras, 2020). Mapping an institution’s research contribution can inform decision-making as it helps to highlight strengths that can drive internal strategic developments at different levels of the institutions from schools, colleges and University wide. As the Goals are a structured and globally recognised framework, they can support research strategies by showing the aim and impact from an institution’s research activities. However operational and other academic strengths, such as teaching, should be also be considered when thinking about the other whole institution strategies, as research is often only one part of the institution’s contribution to the Goals. To achieve the Goals, it is important that the institution engages with the research to ensure that its own policies and practices are in line with the Goals.

Contributions towards the Goals can help with communications and public relations (Jones et al., 2018). Not only can this showcase the positive work that the institution is doing, but it can also spread the awareness of the Goals to the public who might not be exposed to the framework. However, this could potentially turn into ‘greenwash’ if not used correctly and lead to ‘cherry-picking’ certain Goals, instead of showing the Goals as a whole (PricewaterhouseCoopers, 2015).

Along with highlighting strengths, it can also identify knowledge gaps. This is important as the Goals are interdependable and you can’t achieve one goal without considering or impacting other goals. By identifying gaps, this can aid decision-making on whether institutions want to change to improve these gaps. Also, this can provide opportunities for collaborations with other institutions and the local community to help solve these knowledge gaps.

This methodology looks at the research at the end of the journey, however the Goals should also be considered when looking at funding. Funding sources often drive universities’ activities (Ferrer-Balas et al., 2008) so the Goals can help put the impact of a project into a wider context as they are a globally recognised framework. By identifying strengths in relation to the Goals, this can support and strengthen funding bids.

Future development

Teaching

Work to date has centred on mapping research to the Goals, however we have ambitions to broaden our scope to look at teaching. A manual mapping process has been trialled; however, it was laborious therefore utilising our automated tool would be beneficial. In order to map teaching, a new set of keywords will be developed specific to text used to describe teaching. During the manual trial individual module specifications and intended learning outcomes from each degree course were used to select keywords for each of the 17 Goals. We anticipate the full development of SDG mapping for teaching will present some challenges as each university uses different text lengths, style and language to describe degree and module courses. We have ambitions to standardise SDG mapping for teaching regardless of university in the near future.

Operations

Currently the SDG mapping tool is being developed to map the University’s operations. The methodology for this will be similar to that conducted for research and impact instead with operations-specific keyword lists for each of the Goals. This will be achieved using the University’s policies and procedures documents that are publicly available in order to gather operation specific keywords. This new function will provide the building blocks for SDG mapping for business. We believe that businesses are central to achieving the Goals by 2030. Recently, there has been mounting pressure on businesses to report on their sustainability practices but there is not an effective or quick way of doing this (Adams & Zutshi, 2004). Our tool fills the gap in the sustainability reporting market as it uses a robust methodology to map an organisations contribution to the Goals. In order to place more accountability on business organisations regarding their policies and procedures, we are planning to develop a weighted scoring system to assess businesses on whether they perform and achieve the policies they have in place. The standardisation and automation of this process will improve sustainability reporting further and highlight an organisation’s impact within its industry, sector or wider.

Data availability

Underlying data

All data underlying the results are available as part of the article and no additional source data are required.

Extended data

University of Leicester Figshare: Research Excellent Framework Impact Case Study 2014 SDG keywords.

https://doi.org/10.25392/leicester.data.12839444.v1 (Mistry, 2020)

This project contains the following extended data:

• REF impact case study SDG keywords.csv (Keywords for each of the 17 Sustainable development goals derived from REF Impact Case studies 2014. Each goal has a set of keywords associated with the targets and indicators outlined by the UN. These can be used to classify where impact case studies have an impact in relation to the SDGs.)

University of Leicester Figshare: Scopus SDG Search Query codes

https://doi.org/10.25392/leicester.data.12839552.v1 (Mistry & Sellers, 2020a)

This project contains the following extended data:

• Query codes 3.0.docx (Full query codes for each of the 17 Sustainable Development Goals to be used on Scopus advanced search. Useful when searching for publications from specific institutions which match criteria set. Criteria used include, time frame, keywords, and institution)

University of Leicester Figshare: SDG Research Publication Keywords

https://doi.org/10.25392/leicester.data.12839519.v1 (Mistry & Sellers, 2020b)

This project contains the following extended data:

• SDG research keywords.csv (Keywords for each of the 17 Sustainable development goals used to map an institutions published material which is available on databases such Scopus. Each goal has a set of keywords associated with the targets and indicators outlined by the UN.)

Data are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Publisher’s note

This article was originally published on the Emerald Open Research platform hosted by F1000, under the ℈Responsible Management, Quality Education for All, Sustainable Cities and EAUC collection℉ gateway.

The original DOI of the article was 10.35241/emeraldopenres.13881.1

Author roles

Mistry A: Formal Analysis, Methodology, Project Administration, Validation, Writing - Original Draft Preparation, Writing - Review & Editing; Sellers H: Conceptualization, Data Curation, Formal Analysis, Investigation, Methodology, Validation, Visualization, Writing - Original Draft Preparation, Writing - Review & Editing; Levesley J: Conceptualization, Funding Acquisition, Project Administration, Resources, Software, Supervision; Lee S: Conceptualization, Funding Acquisition, Investigation, Project Administration, Resources, Supervision

Funding statement

This work was funded by Mivolve and the grant was assigned to Asha Mistry. This work was also funded by MICRA (Midlands Innovation Commercialisation of Research Accelerator) and the grant was assigned to Asha Mistry, Prof. Jeremy Levesley and Dr Sandra Lee.

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Competing interests

No competing interests were disclosed.

Reviewer response for version 1

Jan Bebbington, Birmingham Business School, University of Birmingham, Birmingham, United Kingdom

Competing interests: No competing interests were disclosed.

This review was published on 30 November 2020.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Recommendation: approve-with-reservations

The main positive aspects of this paper is its focus on linking the SDGs to academic outputs. This is an encouraging development because it highlights the extent to which intellectual activities are supporting the SDGs and provides an example of an approach that a university might use to do this. Likewise, linking impact case studies to the SDGs provides the institution with an idea of how they are supporting these wider aspirations. The development of keywords to support this approach is clearly outlined and should allow broad patterns of coincidence between academic materials and the SDGs to be drawn out.

At the same time, there are some nuances that might be worthy of consideration:

1. If the SDGs are to be seen holistically and if they are to be simultaneously achieved, then this approach is relatively atomistic in its approach and hence might not identify where synergies between cases/outputs/activities might emerge.

2. It would have been good to hear why some of the frequency counts were ‘surprising’ and also if there is an implied ‘standard’ set of patterns that these researchers thought they should see. For example, publication and citation norms of a discipline along with the number of colleagues in each discipline will produce varying maps for each university with the patterns being what they are, rather than signaling anything to be surprised by. If you have a medical school – I would anticipate SDG 3 will feature large. It might also be the case that the foundational SDGs and/or human centred SDGs will predominate, depending on the mix of faculties.

Taken together, this is an interesting paper that will be useful for thinking about the knowledge-SDGs links. It would be difficult to read too much into the patterns from a single paper but there may be something to be said of cross institution and field specific comparisons (should comparison be the goal). Looking at the assemblages of SDGs in particular impact cases and/or outputs might also be a fruitful way forward.

Is the rationale for developing the new method (or application) clearly explained?

Yes

Is the description of the method technically sound?

Partly

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Partly

Are sufficient details provided to allow replication of the method development and its use by others?

Partly

Reviewer Expertise:

Sustainability science within business and management

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

Reviewer response for version 1

Nkeiruka Ndubuka-McCallum, Aberdeen Business School, Robert Gordon University, Aberdeen, United Kingdom

Competing interests: No competing interests were disclosed.

This review was published on 20 November 2020.

This is an open access peer review report distributed under the terms of the Creative Commons Attribution Licence, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Recommendation: approve-with-reservations

I see the usefulness of this mapping tool already. Clearly, SDG 1, 2, 6 are areas that require development. Why might this be the case - no impact case study from the university. It is not a question of not having the required expertise as there are publications in this area. This goes back to the question about the nationalities of the academics working in this area; at least before they naturalised in the UK. Also, the UK may not have the issue of clean water and sanitation (SDG6), but we have seen the fallout of the lockdown informed by COVID. The vigorous campaign for the government to cover and extend free school meals says a lot. While these figures stem from the REF 2014 exercise, what can we expect for REF2021 and the subsequent exercise? These are valid issues that need to be voiced, and there is an opportunity here to interrogate the data. The paper should be reflective of current debates, and that is the main critique I have for this paper. Its relevance is not in doubt, but it can be improved. It is after all about the SDGs; let that come through accordingly.

“‘an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia”.

Provide a relevant in-text citation for the above.

Where direct quotes for the - (Körfgen et al., 2018) is used, include the page number.

Goals

Consider using the Global Goals instead of Goals (see https://www.globalgoals.org/).

But with 17 goals, 169 targets and 232 indicators, it is very complicated to be able to map activities and operations appropriately.

One or more in-text citations is needed to support the above claim. Some studies have discussed the issues around this. Examples to support the claim about the relevance of the Global Goals increasingly being recognised within the HE sector is also needed. It will be useful to be specific since it is not only UK HEIs that are making an effort around this.

Across the UK higher education sector there has been a change in dialogue with new students in the 2019–20 academic year, as both UK and international students recognise the Goals from their school education and are asking how the University is engaging with them and how they can be involved.

The above goes back to the figures from the NUS (2019) survey. Can we have examples of where these students were selected from - i.e. the institutions/universities? It would be interesting to know how this compares across England, Scotland, Wales and Northern Ireland. Briefly discuss/provide an overview of the students - their profile. This information will be useful for universities in their effort to hopefully address these concerns.

With the growing need to rigorously map contributions to the Goals, in line with new reporting and rankings, organisations are looking for ways to map their contributions.

Are there other relevant authors whose works you can draw on to support your work? There seems to be an overuse of Körfgen et al. (2018) - other relevant perspectives would enrich your discussion. A brief discussion of what other institutions have done or the approached employed in an attempt to map the SDGs would be useful for benchmarking purposes and a better appreciation of your proposed method(s).You could explore some universities that are signatories to the Principles for Responsible Management Education (PRME), especially the PRME Champions; the likes of Winchester Business School and Glasgow Caledonian University.

To create an automated solution to mapping academic contributions to the Goals.

Your second objective is focused on harvesting research-related contributions. It would be helpful to know if the last objective is solely related to research - paper publications and associated impacts or is that inclusive of teaching activities?

Mapping the REF Impact Case Studies highlights the very best of research activities that take place within a higher education institution.

I'd be cautious with the use of the term - very best. A degree of subjectivity goes into the evaluation at the university and REF panel level.

Goals. The three highest goals (Goals 3,16 and 10) are mostly related to social sustainability, followed by environmental sustainability (Goals 15, 11 and 13). The Goals with the lowest contribution are Goals 14 and 4.

Can this finding be discussed briefly to add some granularity?Interesting findings. What is the implication of the low contribution to SDG4 for the REF? The teaching-research nexus, does the REF appear to be enhancing or diminishing that unintentionally? The contribution on this front might change given the move to allow teaching-related impacts and associated activities for the REF2021 exercise. Hence, it is worth revisiting.

Whereas other goals that had a high number of publication numbers and citation counts, only have a couple of REF case studies, for example Goals 16 and 10.

What is the possible implication? Arguably, the UK, in comparison to other nations, enjoy a relatively peaceful and just climate. Hence, not an area with many research activities. Is this an area that needs to be picked up by more academics originally from war-torn countries, assuming they are not already doing this?

Number of publications and citation counts relating to each of the SDGs on Scopus 2014–2019.

The figures for SDG4 is interesting. The potential implication for the teaching-research nexus is worth considering.We can say the same for those of SDG14. However, Quality Education (SDG4) is and should be a central agenda for all HEIs within and beyond the UK. The REF should be driving this despite being mostly about research. The possible implications for TEF are also worth highlighting. Both REF and TEF can certainly support universities' commitments and efforts to help achieve the Global Goals.

Table 2. Top 5 SDGs with the highest publication number vs highest citation count.

Concerning the 2nd Rank (Peace, Justice & Strong Institutions), it's interesting that the numbers are not reflective of the low score for impact case studies. Again, what might this be telling us? The critical interrogation of the data aside from the mapping tool developed is equally important. This, I believe, will enhance the quality of the work done thus far before publication. For the 3rd Rank (Reduced Inequalities) - what may be the likely impact of COVID? This is not necessarily about the number of publications, but their direction. More papers baring increased or reduced inequalities, or are we likely to see some communities attain a state of equilibrium on this front? Again, I appreciate the paper is about developing a standardised method for the mapping of the SDGs. However, the interrogation of the data is worth considering briefly. A paper can be harvested from this, providing in-depth insight into what your university is doing towards contributing to the realisation of the SDGs; progress made and areas for improvement. Useful for data for funding bodies too.

However operational and other academic strengths, such as teaching, should be also be considered when thinking about the other whole institution strategies, as research is often only one part of the institution’s contribution to the Goals.

Excellent point! It is reflective of the comment about the teaching-research nexus and REF's intentional or unintentional contribution to this and in what direction? The figures suggest more synergy is needed; teaching-related impact cases studies and pedagogical research are important as applied research. The 5th Rank (Climate Action - citation count) - this too, I envisage will increase as a result of COVID. The national and global lockdown has shown us that the damage done thus far to our planet can be reversed, ideally. At least, we can prevent additional damage for the good of all and future generations; there may be none if we go back to business as usual post COVID.

Figure 2. Number of University of Leicester’s REF 2014 Impact Case Studies relating to each of the SDGs.

11 Impact Case Studies for SDG4 is interesting despite the low number of publications. Can this be discussed a bit - why might this be the case? The keen interest for SDG4 does not suggest that the others are not equally important. However, you can appreciate that SDG4 and SDG9 are central to the achievement of the other goals.

However, this could potentially turn into ‘greenwash’ if not used correctly and lead to ‘cherry-picking’ certain Goals, instead of showing the Goals as a whole.

The above somewhat ties into the summary provided at the start. Aside from the mapping tool developed, critically discuss the findings a bit. The word limit of the journal may not allow an extensive discussion about this, but certainly, another paper can focus on this.

This is important as the Goals are interdependable

and you can’t achieve...

Use cannot instead of can't.

This methodology looks at the research at the end of the journey, however the Goals should also be considered when looking at funding. Funding sources often drive universities'...

Good point. However, the REF certainly can help drive the realisation of the SDGs and thus should be part of the agenda - explicitly stated.

Recently, there has been mounting pressure on businesses to report on their sustainability practices but there is not an effective or quick way of doing this (Adams & Zutshi, 2004).

A more recent citation will further drive the above home.

Is the rationale for developing the new method (or application) clearly explained?

Partly

Is the description of the method technically sound?

Yes

Are the conclusions about the method and its performance adequately supported by the findings presented in the article?

Partly

If any results are presented, are all the source data underlying the results available to ensure full reproducibility?

Yes

Are sufficient details provided to allow replication of the method development and its use by others?

Yes

Reviewer Expertise:

Responsible Management Education (Ethics, CSR and Sustainability), Research Excellence Framework (REF), and Sustainable Development Goals (SDGs).

I confirm that I have read this submission and believe that I have an appropriate level of expertise to confirm that it is of an acceptable scientific standard, however I have significant reservations, as outlined above.

References

1. Ndubuka N, Rey-Marmonier E: Capability approach for realising the Sustainable Development Goals through Responsible Management Education: The case of UK business school academics. The International Journal of Management Education. 2019; 17 (3).

2. Parkes C, Buono A, Howaidy G: The Principles for Responsible Management Education (PRME): The first decade - What has been achieved? The next decade - Responsible Management Education's challenge for the Sustainable Development Goals (SDGs). The International Journal of Management Education. 2017; 15 (2): 61-65

3. Kolb M, Fröhlich L, Schmidpeter R: Implementing sustainability as the new normal: Responsible management education - From a private business school's perspective. The International Journal of Management Education. 2017; 15 (2): 280-292

4. Tourish D, Willmott H: In Defiance of Folly: Journal rankings, mindless measures and the ABS Guide. Critical Perspectives on Accounting. 2015; 26: 37-46

5. Shields R, Watermeyer R: Competing institutional logics in universities in the United Kingdom: schism in the church of reason. Studies in Higher Education. 2020; 45 (1): 3-17

6. Le Blanc D: Towards Integration at Last? The Sustainable Development Goals as a Network of Targets. Sustainable Development. 2015; 23 (3): 176-187

7. McKinley J, McIntosh S, Milligan L, Mikołajewska A: Eyes on the enterprise: problematising the concept of a teaching-research nexus in UK higher education. High Educ (Dordr). 2020. 1-19

8. Haski-Leventhal D, Concato J: The state of CSR and RME in business schools and the attitudes of their students. Third bi-annual Study. 2016.

9. Sangster A: You Cannot Judge a Book by Its Cover: The Problems with Journal Rankings. Accounting Education. 2015; 24 (3): 175-186

10. Macfarlane B: The Morphing of Academic Practice: Unbundling and the Rise of the Paraacademic. Higher Education Quarterly. 2011; 65 (1): 59-73

11. Rasche A, Gilbert D: Decoupling Responsible Management Education. Journal of Management Inquiry. 2015; 24 (3): 239-252

12. McIntosh S, McKinley J, Milligan L, Mikolajewska A: Issues of (in)visibility and compromise in academic work in UK universities. Studies in Higher Education. 2019. 1-12

Figures

Figure 1.

Number of University of Leicester’s publications relating to each of the SDGs on Scopus between 2014 and 2019.

Figure 2.

Number of University of Leicester’s REF 2014 Impact Case Studies relating to each of the SDGs.

Table 1.

Number of publications and citation counts relating to each of the SDGs on Scopus 2014–2019.

	UN Sustainable Development Goals	Publications	Citation Count
1	No Poverty	105	968
2	Zero Hunger	157	6,393
3	Good Health & Wellbeing	656	24,281
4	Quality Education	34	111
5	Gender Equality	155	7,318
6	Clean Water & Sanitation	118	10,823
7	Affordable & Clean Energy	75	1,383
8	Decent Work & Economic Growth	154	1,250
9	Industry, Innovation & Infrastructure	90	1,227
10	Reduced Inequalities	271	6,592
11	Sustainable Cities and Communities	258	10,136
12	Responsible Consumption & Production	128	2,240
13	Climate Action	186	7,820
14	Life Below Water	31	222
15	Life on Land	206	2,612
16	Peace, Justice & Strong Institutions	412	11,490
17	Partnership for the Goals	86	3,778

Table 2.

Top 5 SDGs with the highest publication number vs highest citation count.

Rank	Number of publications		Citation Count
1st	Good Health & Wellbeing	656	Good Health & Wellbeing	24,281
2nd	Peace, Justice & Strong Institutions	412	Peace, Justice & Strong Institutions	11,490
3rd	Reduced Inequalities	271	Clean Water & Sanitation	10,823
4th	Sustainable Cities and Communities	258	Sustainable Cities and Communities	10,136
5th	Life On Land	206	Climate Action	7,820