Evaluating and improving the contributions of university research to social innovation

Abstract

Purpose

The purpose of this study is to assess the contributions of graduate research to social innovation and change for learning and improved transdisciplinary practice. Universities, as centers of teaching and research, face high demand from society to address urgent social and environmental challenges. Faculty and students are keen to use their research to contribute to social innovation and sustainable development. As part of the effort to increase societal impact, research approaches are evolving to be more problem-oriented, engaged and transdisciplinary. Therefore, new approaches to research evaluation are also needed to learn whether and how research contributes to social innovation, and those lessons need to be applied by universities to train and support students to do impactful research and foster an impact culture.

Design/methodology/approach

This paper uses a theory-based evaluation method to assess the contributions of three completed doctoral research projects. Each study documents the project’s theory of change (ToC) and uses qualitative data (document review, surveys and interviews) to test the ToC. This paper uses a transdisciplinary research (TDR) quality assessment framework (QAF) to analyze each projects’ design and implementation. This paper then draws lessons from the individual case studies and a comparative analysis of the three cases on, namely, effective research design and implementation for social transformation; and training and support for impactful research.

Findings

Each project aimed to influence government policy, organizational practice, other research and/or the students’ own professional development. All contributed to many of their intended outcomes, but with varying levels of accomplishment. Projects that were more transdisciplinary had more pronounced outcomes. Process contributions (e.g. capacity-building, relationship-building and empowerment) were as or more important than knowledge contributions. The key recommendations are for: researchers to design intentional research, with an explicit ToC; higher education institutions (HEI) to provide training and support for TDR theory and practice; and HEIs to give more attention to research evaluation.

Originality/value

This is the first application of both the outcome evaluation method and the TDR QAF to graduate student research projects, and one of very few such analyses of research projects. It offers a broader framework for conceptualizing and evaluating research contributions to social change processes. It is intended to stimulate new thinking about research aims, approaches and achievements.

Citation

Belcher, B.M., Claus, R., Davel, R. and Jones, S.M. (2022), "Evaluating and improving the contributions of university research to social innovation", Social Enterprise Journal, Vol. 18 No. 1, pp. 51-120. https://doi.org/10.1108/SEJ-10-2020-0099

Introduction

Contemporary societal challenges are complex, dynamic and urgent. As recognized by the United Nations sustainable development goals (SDG), effective and durable solutions will require broad partnerships between governments, the private sector, civil society and citizens, and must include higher education institutions (HEIs). There is increasing pressure on HEIs to deliver and demonstrate societal impact. For example, university ranking systems, such as the UK’s Research Excellence Framework (REF, 2011, 2021) and Times Higher Education (2020) impact rankings, now include measures of the societal impact of research, and research funding agencies require explicit impact statements. Students also have high expectations about their own contributions. They want to apply their learning and research to solve problems and contribute to positive changes (Willetts and Mitchell, 2016).

The drive for increased impact has catalyzed an evolution in the way research is done, with more interdisciplinary research and transdisciplinary forms of collaboration between researchers, research users and other stakeholders (Nowotny et al., 2001; Hirsch Hadorn et al., 2006). There is growing understanding of the contingency and uncertainty in science, and more recognition that scientific knowledge is necessary but insufficient to generate change; sustainable development involves many normative considerations that link knowledge with action (Sarewitz, 2016; Funtowicz and Ravetz, 1993; van Kerkhoff and Lebel, 2006). Hence, researchers increasingly appreciate that the knowledge and values of stakeholders and intended research users need to be considered and incorporated in the research process (Kasemir et al., 2003).

New problem-oriented and participatory research approaches have evolved to engage system actors in the research process to increase research effectiveness. Variations on these approaches are known as post-normal science (Funtowicz and Ravetz, 1993; Ravetz, 1999), Mode 2 research (Funtowicz and Ravetz, 1993; Gibbons et al., 1994), problem-driven iterative adaption (Andrews et al., 2013), Transdisciplinary Research (TDR) (Carew and Wickson, 2010; Jahn et al., 2012; Klein, 2006; Lang et al., 2012; Pohl et al., 2010; Walter et al., 2007; Wolf et al., 2013) and sustainability science (Brandt et al., 2013; Clark and Dickson, 2003; Kates, 2017; Kates et al., 2001; Roux et al., 2017). These problem-driven, solution-oriented approaches seek to co-produce knowledge and support change through close collaboration between scholars and practitioners (Holling, 1993; Gunderson and Holling, 2002; Clark and Dickson, 2003; Berkes, 2009). The evolution to more engaged, solution-oriented research is evident at all levels, from heightened interest in TDR in graduate student research (Willetts and Mitchell, 2016; Belcher, 2017; Armitage et al., 2019), university-led grand challenge programs (Popowitz and Dorgelo, 2018), international research-for-development programs (Belcher and Hughes, 2020) and the SDGs, which recognize the need to link physical, social and natural capital.

These innovations in the research approach present an opportunity to learn and improve the contributions and influence of more participatory and TDR projects through impact assessment. As TDR approaches are tested across different contexts, there is a natural laboratory to learn what works, where and why. While theoretical approaches are used to evaluate social impact in social enterprise, government and other types of organizations with social responsibilities (Sairinen and Kumpulainen, 2006; McLoughlin et al., 2009; Clifford et al., 2014; Jain et al., 2020), there are fewer documented experiences from assessments of research contributions to social innovation.

HEIs are critical places of knowledge production, perpetuation and dissemination in society, positioning them as key change agents in sustainability transitions (Stephens et al., 2008). New approaches to HEI programming are required to develop TDR competencies (O’Brien et al., 2013; Tam, 2014; James Jacob, 2015; Armitage et al., 2019). Some HEIs have committed to this strategic direction, recognizing many graduate students are keen to make a positive difference in the world; in other cases, HEIs remain an untapped potential as change agents for social innovation (Bloom and Pirson, 2010). HEIs have expanded their missions, adopting new approaches to curriculum development, applied research and training, to equip students to play an active role in social innovation through research (Glasson, 2003; Tam, 2014; Armitage et al., 2019; Bergmann et al., 2021). The scholar-practitioner model has been adopted in some HEIs to increase focus on the practical application of scholarly knowledge and transdisciplinary collaboration (Wasserman and Kram, 2009).

However, there are concerns that HEIs have limited ability to embrace new approaches to education, capacity-building and deliver the research needed to support sustainability transitions (O’Brien et al., 2013). Numerous institutional barriers persist. While there is growing appreciation of the need for transdisciplinary knowledge co-production to address global challenges, incentive and reward structures that sustain disciplinary siloes, a lack of common frameworks and hegemonic concepts and traditions serve to maintain the status quo (O’Brien et al., 2013; Nichols et al., 2013; Benneworth and Cunha, 2015). As one example, traditional disciplinary research assessment criteria for research proposal adjudication, journal article review and other research evaluation disadvantage inter- and TDR (Aguinis et al., 2014; Belcher et al., 2016).

There is an opportunity for HEIs to mobilize research to contribute more and more effectively to social innovation and change through solution-oriented research. To do this, HEIs need approaches and methods to learn whether and how students and faculty research creates social value. HEIs need to leverage lessons learned to recruit, train and support faculty and student researchers and research managers to effectively use research as a tool for social innovation.

This paper contributes to these aims by:

• Presenting a theory-based evaluation method developed specifically for assessing TDR outcomes.

• Applying the method in an outcome evaluation of three doctoral research projects.

• Synthesizing lessons and recommendations to create a learning culture and HEI support for more effective research.

We begin by presenting a research evaluation method developed specifically for engaged solution-oriented research. We then provide overviews of outcome evaluation case studies of three doctoral research projects to illustrate the method and generate lessons regarding how TDR can contribute to change processes. We conclude with lessons about research evaluation and research design and implementation, and provide recommendations for HEIs to help facilitate and support research that contributes to real-world change.

Methodology

Theory-based evaluation

Prevailing research evaluation tools that rely on discipline-defined research quality criteria (Belcher et al., 2016) and bibliometric measures of impact (DORA, 2012; Vanclay, 2012; Hicks et al., 2015) are inadequate for guiding, incentivizing or accurately evaluating research and researchers. With increased focus on impact and the wider application of TDR approaches, there is a greater realization of the need to consider the societal impact (REF, 2011; Penfield et al., 2014; Sarewitz, 2016; Aguinis et al., 2014) and use a broader range of research evaluation tools.

We use a theory-based approach that builds on program evaluation methods including outcome mapping (Earl et al., 2001), payback framework (Buxton and Hanney, 1996) and contribution analysis (Mayne, 2001, 2012). The method uses a detailed project theory of change (ToC) to model the change process. The ToC explains both how and why a project is expected to contribute to a set of results (i.e. outputs, outcomes and impacts) (Coryn et al., 2011; Belcher et al., 2020; Vogel, 2012; Weiss, 1997). The ToC details the primary actors, steps and pathways in the change process and specifies the theoretical reasons for the changes. A well-specified ToC provides hypotheses about each step in the change process that can be tested deductively using empirical evidence to answer the question: who is doing what differently as a result of the research?

As we assess whether and how research contributes to change, we need a way to characterize each project. Belcher et al. (2016) developed a TDR quality assessment framework (QAF) based on a systematic review of literature defining and measuring quality in interdisciplinary and TDR. The 27 criteria are organized under four principles, namely, relevance, credibility, legitimacy and effectiveness [1] (Figure 1 in Belcher et al., 2016 for criteria definitions). The QAF serves as a checklist of project elements that, in theory, should contribute to societal impact.

Each outcome evaluation case study used the same methods detailed in Belcher et al. (2020).

Theory of change documentation

The first step was to document each project ToC. This was done in a workshop with the doctoral student researcher, where we modeled the ToC (i.e. project activities; outputs; key actors; intermediate, end-of-project and high-level outcomes; and intended impacts), documented underlying assumptions and identified potential evidence to empirically test the ToC.

Data collection

Data were collected by reviewing relevant project documents (e.g. project proposals, dissertations and workshop notes) and public documents (e.g. government documents, non-governmental organization publications and press releases) and by interviewing or surveying people with knowledge of the project and the system in which the project intervened. An initial set of informants was identified in consultation with the student researcher, with additional informants identified through snowball sampling. Informants included project supervisors, collaborators and research assistants and other research participants. Semi-structured interviews used a funnel design, starting with general questions about the respondent’s role, experience and perspectives on the context, the social problem and its evolution and then focusing on their assessment of the project and its contributions (Appendix 1 in the supplementary materials for a sample interview guide). In some cases, online surveys were used to get a broader range of perspectives.

Outcome assessment

Data were coded in NVivo according to the outcomes specified in the ToC. Coded data were assessed to determine whether each outcome had been realized. We then evaluated the project’s contributions to each outcome for evidence of causal linkages between outputs and outcomes. We also asked informants directly if they could link the project to changes they observed.

Assessing the strength of project contributions

A key challenge in assessing the impact of any intervention in a complex system, with multiple interacting processes, actors, time lags and feedback loops, is that it is impossible to specify a counterfactual. With a single case, it is not possible to use statistical comparisons (i.e. experimental or quasi-experimental methods). In lieu of a reliable counterfactual, we considered and tested competing hypotheses for how a change may have happened, leaving room to consider additional factors and alternative explanations for how the project may or may not have affected the outcome in question (Donaldson, 2009; Hitchcock, 2018; Rossi and Freeman, 1989; White, 2009). For example, there may be other projects or agents working toward similar outcomes. Interviews with informants include explicit questions about whether, in their expert judgment, observed outcomes could have been realized in the absence of the project.

Quality assessment framework scoring

Four evaluators reviewed project documentation and interviews prior to scoring. Each evaluator scored the criteria independently on a scale of 0–2 against the project’s purpose. A score of 0 was awarded if the criterion was not satisfied; 1 if the criterion was partially satisfied; and 2 if the criterion was completely satisfied. After scoring independently, evaluators met to discuss scores, justifications and evidence for each score. Averages were calculated for final scores. The scores indicate TDR characteristics that were strong, present but incomplete or absent in the project.

Deriving change mechanisms

Like traditional disciplinary research, TDR aims to create new knowledge through data collection and analysis, methods development and theory testing. By design, TDR also supports social processes. Mechanisms were identified through the coding process, as data evidencing how outcomes were realized can often be linked to specific design characteristics, project activities, engagement approaches or project contributions. Previous comparative analyses of research-for-development projects (Belcher et al., 2019) identified several kinds of mechanisms that help explain observed changes. We applied that set of mechanisms in this analysis using the evidence available from interviews and document review to determine the mechanism(s) relevant for the change being evaluated.

Comparative analysis

Each case study is documented in a detailed outcome evaluation report (see Claus et al., 2020 for an example). The cases were then compared based on the extent of intended outcome contributions and the strengths and weaknesses of their design and implementation to elicit lessons to inform more impactful research in the future.

Case studies

Royal Roads University’s (RRU) Learning, Teaching and Research Model has an explicit mission to teach and generate research that contributes to transformation in students and the world (RRU, 2020b, 2020c). Programs encourage the study of complex real-world problems using interdisciplinary approaches to problem-solving for organizations, communities and society (RRU, 2020a). The Doctor of Social Sciences (DSocSci) program attracts mid-career professionals and encourages students to focus their research on social or environmental problems derived from their professional experience (Pulla and Schissel, 2017; RRU, 2020a).

Cases were selected through a systematic review of graduate research housed in an online repository. We applied seven selection criteria:

1. A clearly stated problem/issue.

2. A socially relevant research question.

3. Inclusion of community or other stakeholders.

4. An articulation of how the project would lead to expected outcomes (implicit or explicit ToC).

5. Appropriate research design and application of methods.

6. Conclusions with demonstrated potential for outcomes (e.g. provides applicable recommendations).

7. Completed within five years of primary data collection.

This paper discusses three doctoral projects, which are detailed in Table 1. All projects were completed as part of the DSocSci program at RRU, conducted by mid-career development practitioners, with case studies and fieldwork in an African context and focused on topics within the realm of international development.

Truth-telling in Uganda

Armed conflict in Northern Uganda between the Lord’s Resistance Army and the Ugandan government affected over 100,000 young people who were abducted, internally displaced, born in captivity and/or disabled by the war (Annan et al., 2008; UNOCHA, 2004). In 2006, Article 3 of the Juba Agreement marked the conclusion of the atrocities and began a transitional justice (TJ) process to support peacebuilding and reconciliation. Individual statement-taking is a common method for truth-telling processes (UNICEF, IRC, and ICTJ, 2010); while the approach has value, it has the potential to re-traumatize victims (Hamber, 2009; Hayner, 2011). There is a need for improved ways to engage young people, not only to gather a full and impartial record of past events but also to support healing and reconciliation. The researcher sought to fill this gap using participatory action research in northern Uganda to explore young people’s perspectives on post-conflict truth-telling processes.

The central aim of the Truth-telling Project (TTP)’s ToC (Figure A1 in Appendix 2) was to ensure meaningful and ethical engagement of young people in issues that affect them. The project was designed to engage conflict-affected youth as equal partners. It aimed to contribute to outcomes through five interconnected impact pathways:

1. Empower young people by building participants’ capacities to participate in truth-telling commissions.

2. Contribute to improved policy and policy development processes.

3. Develop partners’ organizational capacity and practice to work with young people and in TJ contexts.

4. Contribute to the academic discussion and trajectory of research.

5. Support professional and career development of the student and Ugandan research assistants.

Philanthropy in Tanzania

Private development aid (PDA) plays a growing role in the development arena, but the influence of philanthropic giving and how it can be effectively managed, monitored and enhanced is under-researched. Despite claims that PDA contributes to international development efforts (e.g. poverty reduction, health, sanitation, etc.), empirical evidence is scarce. The Private Development Aid Project (PDAP) addressed this gap with an exploratory case study of Moshi, Tanzania, where a high concentration of private development activity has emerged in recent decades. The purpose of the project was to empower communities through the transformation of the management and practice of PDA organizations. The project focused on understanding recipients’ experiences of PDA, both end-users whose lives are impacted by private aid (i.e. end-use recipient) and organizations that receive philanthropic funds (i.e. project implementers). The stated aim of the PDAP’s ToC (Figure A2 in Appendix 2) was to provide a foundation of knowledge on the perceived nature, reach and influence of PDA as experienced by recipients of philanthropic funds. This knowledge was intended to inform recipient organizations and donors via recommendations to enhance PDA implementation and benefits for end-use recipients. The project aimed to contribute to four impact pathways, to:

1. Contribute the academic discussion.

2. Inspire organizational change to encourage “thriving” (Funk, 2016, p. 9) practices.

3. Empower participants.

4. Support the student’s professional development.

Sanitation in the Niger Delta

Approximately 50 million people practice open defecation in Nigeria, with slow progress in water, sanitation and hygiene (WASH) in the Niger Delta (Gilbert, 2017). Nigeria was listed among countries “not on track” to fulfill the sanitation objectives of the seventh millennium development goal (MDG). Improving access to adequate WASH is critical for community health and development. The Niger Delta region requires tailored technology options to address the WASH gap because of its low altitude coastal geography, high annual rainfall, high-water table and frequent flooding. The region also has challenging social conditions, including low levels of education, income and cultural norms and attitudes that impede the adoption of improved WASH practices. The Sanitation in the Niger Delta Project (SNDP) ToC’s (Figure A3 in Appendix 2) purpose was to improve the health, social and economic development of communities living in the Niger Delta. By gaining insights into the cultural, social and economic context, the project aimed to guide and inform approaches, including technological options, for sustainable sanitation in the Niger Delta’s riverine communities. The research aimed to contribute to this through three interconnected pathways:

1. Influencing the capacity and practice of the WASH sector to improve community development.

2. Government policy.

3. Professional development.

Results

Outcomes realized

Table 2 provides a summary overview of the impact pathways through which each project intended to contribute to change, and lists the key outcomes realized for each case study. Not all intended outcomes were realized. Detailed results of the outcome assessments can be found in the supplementary materials (Figures A1–A3 in Appendix 2 for ToCs depicting outcome realization and Tables S1–S3 in Appendix 3 for detailed assessments). Evidence indicates the projects made varying contributions to changes in practice, capacity, research and policy.

Truth-telling Project.

The TTP contributed to the realization of outcomes for young people, partnering organizations, the research team and international, national and sub-national policy. All intermediate and end-of-project outcomes in each pathway were realized, and there was some progress toward higher-level changes (Figure A1 in Appendix 2). Participating young people were engaged as equal partners, co-designed a process that respected their wishes and needs and experienced first-hand an example of how a TJ engagement process could be conducted. Capacity-building was central to the project. Participants developed new social and communication skills, new knowledge of truth-telling commissions and TJ processes, attitudes toward the value of their perspectives in post-conflict truth-telling and new relationships with other participants.

Changes in partner organizations’ capacity and practice were realized through collaboration and partnerships that supported needs, and co-produced knowledge to realize mutual benefits of peacebuilding in Uganda (Figure 1). The main partner organization increased their focus on working with young people as a result.

Changes in policy were supported by increased attention by partner organizations to the issue, having a piloted process and participating young people’s capacity to engage in a policy process. The partner organization brought government stakeholders, civil society organizations and young people (including several that participated in the project) together to contribute to a national policy on TJ.

The project enabled the researcher and local research assistants to develop research capacities and work with young people in a TJ context. The experience exposed the research team to new networks and supported their interests in continuing work on TJ and young people. Both research assistants have continued careers in the field. The research team’s publications have been cited to support arguments for child learning through active interaction, noting PAR as an appropriate method, and young people have the right to contribute to issues that affect them. The researcher has continued in a scholar-practitioner capacity, making contributions to the academic discourse and advocating young people’s engagement in climate change and healthcare issues. The transferability of experiential learning from the project has supported progress toward high-level outcomes for young people beyond the Ugandan TJ context.

Private Development aid Project.

The PDAP realized most of its intermediate and end-of-project outcomes (Figure A2 in Appendix 2). The project supported changes in knowledge of academics and practitioners, as well as capacity-building of participants, PDA practitioners in Moshi and globally and the researcher. The project also contributed to changes in the researcher’s professional relationships. There is evidence of changes in PDA practice through uptake and application of PDAP recommendations at the individual and organizational levels.

Evidence of outcome realization was strongest for actors directly engaged by the project, such as recipient organizations and end-use recipients. Individuals built both personal and professional relationships with the researcher. Participants learned about “floundering” (Funk, 2016, p. 9) practices and co-produced the recommendations on how to be a “thriving” organization. As co-producers, participants had both ownership and agency over the findings, which increased the likelihood that they would apply project knowledge. Participants’ involvement also prompted personal reflection on current practices and decision-making on whether changes needed to be made. For many participating organizations, the findings affirmed their approach and direction. For others, the findings helped identify strategic changes, such as a formal reassessment of practices, changing proposal and reporting protocols and planning for future leadership changes.

Other Moshi-based non-participating organizations benefitted from the project’s work. The researcher intentionally planned and implemented activities to share knowledge, build capacity and support improved practice in the sector. The researcher offered pro bono consulting (e.g. focus groups on organizational management and strategic planning sessions), which was open to all during the fieldwork period, and many organizations used these services. This enabled the researcher to share preliminary project insights and offer contextually relevant guidance to organizations operating in the sector.

Outcomes in the research pathway were less strong. While the researcher made an original academic contribution and researchers gained awareness of the findings via international conferences, evidence of uptake is low. There are indications of the intended application of the findings by two researchers in the student’s sphere of influence, but the research has not been widely cited. As the researcher did not make further academic contributions through peer-reviewed publications, some informants felt this was a missed opportunity. There is, however, evidence of project influence on undergraduate students. Several professors from two Canadian universities invited the researcher to guest lecture. Students attending these lectures learned about international fieldwork, applied qualitative research, the African development context and the philanthropic sector.

The project also influenced the researcher’s professional development. They used the doctoral project as an opportunity to investigate questions they encountered as a practitioner. The researcher enhanced their facilitation and consulting skills, research capacities and analytical thinking. Moreover, with doctorate qualifications, the researcher is increasingly recognized as a PDA expert and scholar-practitioner. The researcher has integrated and applied knowledge from their doctoral experience in their consulting work in Tanzania, Canada, the US and beyond. Through the doctoral program, the researcher was invited to mentor another DSocSci student. This mentorship later evolved into a professional collaboration whereby both work on the executive board of an organization established by the mentee.

Sanitation in the Niger Delta Project.

All SNDP intermediate and end-of-project outcomes were partially or fully realized (Figure A3 in Appendix 2). Outcomes pertaining to the knowledge, attitudes, skills, relationships and behaviors of the local WASH sector, Partnership Initiatives in the Niger Delta (PIND) staff and communities were realized through the involvement of these actor groups in the research process. As an exploratory project, the SNDP identified key knowledge gaps and helped stakeholders become aware of WASH gaps, issues and opportunities through targeted dissemination of the findings. To support data collection but also eliminate silos and bridge networks in the sector, the researcher created a WhatsApp group connecting local WASH practitioners, which is still in use, to share knowledge and best practices and expose them to diverse perspectives and experiences. Through close involvement in the research, participants and PIND staff reflected on their own practices. Working at PIND, the researcher influenced and ensured project findings and regionally appropriate sanitation technologies were on display at PIND’s demonstration center. Subsequently, visitors who viewed the exhibit were exposed to the information and gained more awareness about WASH issues and solutions.

Intermediate and end-of-project professional development outcomes were realized and focused on the researcher’s knowledge, skills and networks to support their future work on WASH and community development. By leveraging the researcher’s professional position at PIND, knowledge gained through the project was successfully transferred to their colleagues to increase capacities. The project reinforced the researcher’s passion for WASH, leading the researcher to establish an organization to continue this work internationally using a community-based approach. The project provided an opportunity for the researcher to further develop their skills, increase professional exposure and become recognized as an expert by practitioners working on sanitation in the Niger Delta and beyond.

It is too early to expect changes in government policy and practice. However, government actors’ participation in data collection and the WhatsApp group may have stimulated reflection on WASH policies and practices.

Project design and implementation

Each project had some transdisciplinary elements, but only the TTP satisfied all TDR criteria (Figure 2). This kind of analysis could be used to identify strengths and weaknesses in student research, and inform curriculum design to support improved research planning and implementation. Case-specific QAF results are discussed below (Tables S4–S6 in Appendix 3 for more detailed scores and justifications).

Truth-telling Project.

The TTP applied all transdisciplinary principles (Figure 2). The project facilitated meaningful engagement of young people who experienced the Ugandan conflict, and provided them an opportunity to share how they wished to be involved in future TJ processes. All project activities were designed with this purpose in mind. Genuine and explicit inclusion of participants with the intention to build their capacity was integral to empowering young participants. Interview respondents described the project as highly ethical, participatory and beneficial to the young people involved. The project focused on understanding and accommodating the unique intersection of TJ for young people in the Ugandan context. The project explicitly identified and planned for outcomes from the start. Inspired and driven by the need for meaningful and ethical engagement of young people on issues that affect them, the project included young people as partners in decisions around how the project progressed. The project was lauded for its appropriate application of PAR methods and collaborative engagement with strategic partners to build capacities of participants and partnering organizations. The researcher was well-equipped to carry out the research, given diverse experience with research, working with young people and the Global South. This case study was well-designed and implemented, achieving full QAF scores.

Private Development Aid Project.

The PDAP exhibited varying degrees of transdisciplinarity, satisfying most QAF criteria (Figure 2). With clear identification of research entry points within Moshi’s PDA sector, in combination with professional insights and engagement in similar problem contexts in Africa, the researcher designed and implemented a relevant and appropriate research project. The design was feasible owing to adequate funding support and the researcher’s prior experience. The project was designed to influence change; while intended outcomes were documented, the underlying logic, assumptions and ideas on how the project would exert influence were underdeveloped or implicit. The project was guided by clear research questions, sufficiently broad preparation of academic and grey literature review, multiple rounds of fieldwork and methods that were fit to purpose yet adaptable to the range of participants. The project’s credibility could have been strengthened if it had clearer objectives, stronger methodological and epistemological integration and a more complete presentation of results and limitations. Trust-building played an important role in supporting inter-personal trust between the researcher, participants and the wider PDA community in Moshi, as well as trust in the project and its findings. In addition to obtaining an ethical review from RRU’s research ethics board, the researcher sought and received approval from the Tanzanian Commission for Science and Technology to conduct the fieldwork. However, other key aspects of legitimacy were weaker. While the researcher disclosed and reflected on potential bias and their own positionality, these elements were brief and not discussed in terms of their effect on the findings. Several project activities were inclusive and collaborative in design (e.g. multiple rounds of interviews to validate findings and co-generate recommendations, pro bono consulting); however, there was unrealized potential to organize formal collaborative arrangements with participating organizations. Nevertheless, several aspects of the project helped make it effective. The project uncovered and filled relevant practice gaps and the findings had practical application. There is evidence that some participating organizations and practitioners not engaged directly in the project have applied learning to their work.

Sanitation in the Niger Delta Project.

The SNDP satisfied some transdisciplinary criteria (Figure 2). The researcher’s previous experience working in the Niger Delta supported the appropriate implementation of the project, as they had in-depth knowledge and understanding of the sector and could leverage their professional connections. According to informants, this helped generate practical and timely recommendations and enhanced project influence. The SNDP effectively addressed a socially relevant research problem and effectively engaged with the problem context by aligning the project with MDG 7. The researcher was well-positioned to influence change at PIND and access Nigerian WASH networks. However, criteria relating to relevant research objectives and effective communication were not fully satisfied. This likely affected outcomes relating to the uptake of SNDP findings by members of the WASH sector outside of the researcher’s direct sphere of influence. For example, had the findings been shared through targeted communications with government actors, it is possible that more government WASH practitioners and policymakers would be aware of the issues and available solutions to enhance governmental commitments to WASH. The project followed RRU ethical review protocols. However, the SNDP was less thorough in disclosing potential bias and PIND’s corporate connections. There was scope for the project to be more collaborative through the researcher’s professional connections at PIND and with other WASH practitioners to improve the uptake and application of findings. Effectiveness manifested most clearly in the project’s contribution to a significant outcome. Informants indicated that they now think more deeply about the topic and related issues, as the project contributed to the local knowledge base. Owing to limited previous research on WASH in the Niger Delta, some informants felt the project provided a basis for further exploration on the topic. However, others suggested the SNDP summarized what was already known in the sector and did not contribute to solution development. The project effectively built the researcher’s capacities and professional development, but missed the opportunity to more intentionally integrate capacity-building opportunities for other system actors into the project’s design.

Mechanisms

The term “mechanisms” is used to refer to the underlying social, psychological or economic reasoning or stimulus that causes an individual or group of actors to take a particular action. Figure 1 describes the mechanisms through which each case study realized outcomes in each pathway. The TTP leveraged the greatest diversity of mechanisms to realize outcomes across the greatest number of impact pathways. The PDAP and SNDP leveraged relatively fewer mechanisms across fewer pathways. PDAP used the same four mechanisms across its respective impact pathways to support different outcomes, demonstrating the multi-functionality of a singular mechanism and how one mechanism can be leveraged in different ways. The SNDP leveraged mechanisms of change more diffusely.

Overall lessons

Graduate research contributions

Our findings show that graduate students can make diverse contributions to change processes through their research. Influence on multiple system actors requires a diversity of mechanisms, and therefore, projects need to be strategically designed and implemented to influence change. These findings corroborate Belcher et al.’s (2019) conclusions that projects with more transdisciplinary qualities can use more diverse mechanisms and exert greater influence across more impact pathways.

All case studies aimed to influence organizational practice, with varied success. The TTP selected local partners for their propensity to value the ethical engagement of young people in truth-telling to strengthen collective action. Project activities targeted partner organizations’ learning and skill development to engage young people, with the assumption that equipping partners would sustain progress toward higher-level outcomes following project conclusion. Project relations functioned on a foundation of common understanding, trust and reciprocity. The PDAP was able to foster more “thriving” organizational practice through a combination of interview-prompted reflection and free consulting services. For some organizations, the project affirmed successful aspects of their current direction. Others were inspired to reassess and adjust their practices. The SNDP leveraged the managerial position of the researcher at PIND, who transferred learning to colleagues to support staff effectiveness. PIND staff working alongside the researcher in the demonstration center and trainers delivering the WASH training curriculum benefitted most directly from this learning. However, PIND discontinued their focus on WASH after the researcher left the organization. Therefore, PIND did not fully leverage opportunities presented by the project to support the future realization of outcomes, suggesting that insufficient organizational capacity was developed.

Projects had varying degrees of transdisciplinarity, but there were strong TDR elements common to all. Each project carefully identified and addressed a socially relevant problem. This was supported by each researcher’s prior professional experience and knowledge. All researchers expressed an interest to make a difference, and had relevant experience and networks that could be drawn upon to effectively complete their research. All projects were subject to University ethics review. The TTP’s participatory design, post-conflict context and engagement of young people required extra caution and attention to ethical principles to build legitimacy. The project was effective as it emulated a process that future truth-telling commissions involving young people could replicate.

The QAF appraisal also highlighted differences between projects’ design and implementation that explain discrepancies in outcome results. Developing an explicit ToC entails designing activities for outcomes. The TTP was guided by the intention to contribute to meaningful outcomes for young people, and strategically planned activities around these intentions. Project activities emulated the purpose of meaningful engagement of young people on issues that affect them. This was made explicit, and guided team thinking around design and implementation. Opportunities were created and realized through a co-generated and co-owned research process. While the PDAP and SNDP had implicit ideas about how the research would influence changes in their respective contexts, neither ToC was fully developed nor used as part of project planning and implementation. These results demonstrate how planning for outcome realization using ToC supports strategic project management and social value creation if approached with adequate reflection, critical thinking and honesty (Vogel, 2012; Weiss, 1997).

Networking, peer-learning and knowledge co-production with intended beneficiaries are necessary to support change processes for sustainable development (Holling, 1993; Gunderson and Holling, 2002; Clark and Dickson, 2003; Berkes, 2009; Hazenberg et al., 2019). The extent of genuine and explicit inclusion was another distinguishing quality between projects. The TTP was designed as a highly participatory project to empower its participants as decision-making partners. The ethical design ensured inclusion was genuine, and the needs of TTP participants were considered and integrated through adaptive project management. In the PDAP, invitations to participate or receive pro bono consulting were open to all organizations in Moshi. This enabled inclusion of local Tanzanian, ex-patriate and foreign implementer perspectives, end-use recipients whose voices are often excluded, and private donors’ input in the project. The project undertook a second round of interviews to validate data and involve participants more actively in the findings; for some, this enabled knowledge co-generation, whereas for others this interview approach remained an extractive process. The SNDP placed less emphasis on capacity- and relationship-building with research participants, and data collection was perceived to be extractive. Intentional collaboration and knowledge dissemination with system actors, organizations and boundary partners were suggested to further support the implementation of SNDP’s recommendations.

Higher education institutions training and support

All case studies had a common thread of professional development for the researchers (and if applicable, others on the research team) that supported outcomes and stimulated further progress toward higher-level changes. RRU programming supported professional development in all cases. Gaining insights, skills and accredited expertise through the doctoral experience were explicit goals for all researchers. The scholar-practitioner model (Wasserman and Kram, 2009) helped build the relevance necessary for projects’ outcomes. The DSocSci program allowed all researchers to pursue research that was relevant to their practitioner roles and fit with their career aspirations.

The DSocSci is a blended program, which admits students from diverse professional backgrounds, balancing online coursework with annual on-campus residencies. For all researchers, this blended structure enabled them to continue working while studying and doing fieldwork. As a result, much of the student research done in the program is derived from real-world problems and applied to real-world contexts.

These cases illustrate the potential for HEIs to facilitate interdisciplinary programs and learning cohorts that enrich student experiences, perspectives and ultimately research. The DSocSci program fostered an open space where the cohort could be a sounding board for ideas during the projects’ design phase. The PDAP researcher found the cohort useful to understand broader perspectives of their research problem and challenge internalized assumptions. The program’s interdisciplinary focus encourages integrative approaches to problem-solving and exploration beyond disciplinary or academic boundaries. The interdisciplinary environment supported the researchers to influence change as scholar-practitioners, as the program encourages blending the practitioner and community settings with the academic realm (not often encouraged at traditional universities).

Cohorts also can facilitate networking between students, enabling innovation, mutual learning and new collaborations. In one case, faculty in the DSocSci program linked two of the researchers from different cohorts together, initially as a mentor-mentee relationship, which later resulted in a serendipitous professional partnership following the program. Upon connecting over their shared interests and professional backgrounds in development, the researchers uncovered further parallels between each of their respective doctoral research projects and the objectives of the organization for which they now both work.

Outcomes in the research pathway were a lower priority for the three projects (one project did not intend to influence research at all). All researchers contributed to the academic knowledge base on their respective topics through their dissertations, and one researcher published a peer-reviewed article that received some academic engagement by other researchers (e.g. TTP). However, students often need significant support from their supervisors, professors and university (e.g. access to library resources following completion of degree) to meet peer-reviewed journal requirements to have influence in the academic arena. Without such support, the PDAP researcher was unable to publish more from their research beyond their dissertation, despite being highly motivated. This is an area where HEIs can consider supporting student research for mutual benefits.

Conclusion

These cases demonstrate that graduate research can contribute to change processes in many ways. The projects aimed to influence government policy, organizational practice, other research and the students’ own professional development. All contributed to many of their intended outcomes, but with varying levels of accomplishment. Projects that were more transdisciplinary had more pronounced outcomes. Process contributions (e.g. capacity-building, relationship-building and empowerment) were as or more important than knowledge contributions. Moreover, there is evidence that all projects stimulated change processes that will continue.

Graduate students seeking to make a difference with their research should consider using transdisciplinary approaches when designing and implementing thesis/dissertation projects. This means taking the time to fully understand the social problem and its context, identifying key actors and processes and strategically designing research to be useful and used. Developing an explicit and detailed ToC has proven to be a key ingredient of effective change-making research.

HEIs can promote effective student research by encouraging and supporting solution-oriented TDR. This starts with recruiting students who are interested in change-making and providing appropriate training in epistemology, methodology and stakeholder engagement. Training and support in the use of ToC in project design and implementation are critical. Research projects should be guided by considerations of who will do what differently as a result of their research and why. Experience from HEI training in ToC shows some promising results for building TDR competencies; applying ToC in project planning is a useful exercise to place the project within a broader sustainability context to build relevance and develop a systems perspective (Armitage et al., 2019). Social impact measurement needs to be embedded in project management decisions (McLoughlin et al., 2009). HEIs should encourage the uptake of adaptive project management principles and tools to ensure students are well-positioned to contribute to and create change through their research. Students also need to be guided to design their projects to be feasible. TDR can be more complex, unpredictable and challenging than disciplinary research. It is important to recognize the limits of any project, set reasonable objectives and be prepared to adapt as needed (Bergmann et al., 2021).

Finally, there remains much to learn about how to do more effective research to support social transformation. Novel research approaches are being tested in various contexts, and we need to take advantage of the opportunity to learn what works, where, how and why. Our methods are based on frameworks that can be adapted for use in TDR planning, adaptive management and evaluation for impact. If effectively mobilized, these frameworks could help reduce institutional barriers that prevent HEI research from reaching its full potential for scientific and social impact (Jahn et al., 2012; O’Brien et al., 2013; Nichols et al., 2013; Benneworth and Cunha, 2015). HEIs need to cultivate an impact culture that includes more systematic impact assessments to ensure continuous learning from successful research endeavors and reflecting on failures (Fam and O’Rourke, 2020). Attention is needed at all phases of the research cycle to guide and check that: impact strategies are considered in research design; researchers reflect on and adapt to changes and learning that may affect a project ToC; there is sufficient evaluation of outcomes; and lessons are systematically learned and used for continuous improvement.

Figures

Figure 1.

Mechanisms leveraged by each case, organized by pathway

Figure 2.

QAF scoring diagrams of (a) relevance, (b) credibility, (c) legitimacy, and (d) effectiveness for the three case studies

Figure A1.

TTP Theory of Change, with outcomes colour-coded to reflect the extent of outcome realization (Claus et al., 2020)[4]

Figure A2.

PDAP Theory of Change, with outcomes colour-coded to reflect the extent of outcome realization[5]

Figure A3.

SNDP Theory of Change, with outcomes colour-coded to reflect the extent of outcome realization[6]

Table 1.

Case study overview

Case study	Research topic (location)	Intended influence	Data collected for evaluation
Truth-telling Project (TTP) (2012–2014)	Young people’s engagement in post-conflict truth-telling commissions (Uganda)	• Empowering Ugandan young people • Building capacity in the local research team and partner organizations • Functioning of local truth-telling commission processes in Uganda • National policy processes in Uganda • International policy on truth-telling commissions • Research agendas on the meaningful and ethical engagement of young people • Professional development of the PI	• Document review of 24 project and external documents • 19 semi-structured interviews • Survey of 35 project participants • Three media (video and websites) • Altmetrics of tailored products
Private Development Aid Project (PDAP) (2014–2016)	Recipient perspectives of private development aid (Tanzania)	• Practice of private development aid organizations in Moshi • Coordination of Moshi’s aid sector • Research on private aid • Professional development of the PI	• Document review of 21 project and external documents • 23 semi-structured interviews • 15 media (video, websites and news posts) • Altmetrics of tailored products
Sanitation in the Niger Delta Project (SNDP) (2015–2017)	Water, sanitation and hygiene (WASH) in riverine communities of the Niger Delta (Nigeria)	• WASH practice of WASH sector in the Niger Delta • WASH practice and culture of Niger Delta communities • Local government policy on effective WASH • Professional development of the PI	• Document review of 7 project and external documents • 15 semi-structured interviews • Survey of 42 WASH professionals • 3 media (video, websites and news posts) • Altmetrics of tailored products

Note:

PI = Principal investigator

Table 2.

Outcomes realized by each project’s influence pathway

	Intended influence pathways
Case	Organizational and sectoral practice	Professional development	Research	Empowering research participants	Government policy and practice
TTP	Partners: • recognized the value of young people’s voices and engagement • applied and shared creative methods in their work • enhanced commitment to working with young people	Research team (including local research assistants and PI): • developed new research skills • gained recognition for their expertize • continued careers in TJ or working with young people	Researchers: • used the research to take up new questions and adapt approaches	Participating young people: • gained social and communication skills • increased confidence to share with adults • recognized the value of their voice • learned about truth-telling commissions and TJ processes in other jurisdictions • developed relationships with other research participants and partners • were involved in drafting a national policy on truth-telling commissions	National governments and intergovernmental organizations: • developed new policies, frameworks and recommendations that integrate young people’s perspectives
PDAP	Moshi PDA organizations: • gained knowledge and skills to better manage PDA practices (via pro bono services offered by the PI)	PI: • gained diversified and context-specific PDA knowledge and insights • gained and enhanced their skills (e.g. facilitation, research skills and analytical thinking, organizational governance and management, leadership, monitoring and evaluation, etc.) • is recognized as a PDA expert • applied knowledge from doctoral experience to international consulting work • built a personal relationship with a DSocSci student (via mentoring), which evolved into a professional collaboration	Researchers: • became aware of the research via conference presentations • invited PI as visiting fellow at a local university • indications of the intended use of PDAP findings by two researchers in PI’s sphere of influence Undergraduate students: • Gained awareness via guest lectures at two local Canadian universities	Individual participants: • built trust and relationships (both personal and professional) with the PI • gained and co-produced knowledge about “thriving” organizations • reflected on current practices • applied PDAP findings when transferring to a new organization Participating organizations: • gained affirmation of their practices and current organizational direction • changed practices based on PDAP findings and recommendations	n/a
SNDP	Local WASH sector: • has a forum for WASH discussion • learned about and recognized WASH gaps, issues and opportunities PIND staff: • gained WASH knowledge, skills and capacities • applied new knowledge and skills in related work Communities: • visitors to ATED have increased knowledge and more awareness about WASH issues National WASH sector: • has capacity for enhanced knowledge sharing and learning (via networks linked by project)	PI: • professional development enhanced by research experiences • gained a deeper understanding of issues in WASH and how to approach working in challenging contexts • continues work in Niger Delta WASH sector and beyond	n/a	Research participants: • reflected on their WASH practices • recognized value of research to inform practice	Local governments: • intended outcomes not realized

Table A1.

General questions about the respondent, their expertise on the topic and recent/significant changes in topic (purpose to build rapport and clarify the context)

Main question	Probes	Intent: what we are trying to find out Do NOT ask these directly
1. What is your role within [organization]?	• How is your work related to [topic]? • How long have you been doing this kind of work?	Understanding the respondent’s job/organization and the relevance of the topic to their work Finding out the expertise of the respondent and their professional connection to the topic, as well as their influence on the topic of focus
2. What role does [organization] play in young people’s engagement in post-conflict truth-telling/transitional justice processes?	• How long has your organization been involved in work related to [topic]?
3. What are the main challenges related to young people’s engagement in post-conflict truth-telling/transitional justice processes?	• What is the reasoning for these challenges?	Personal expertise and perceptions on the topic of focus Interviewee’s knowledge level, understanding, and perceptions on the problems and issues relevant to the focus of the project – what do they think the problems are and how they frame the problems QAF: Rel1, Rel2, Rel3, Rel5
4. What have been the most important developments related to young people’s engagement in post-conflict truth-telling/transitional justice processes in the last five years?	• In the discussions, events, ideas, institutions, policy, and/or practice?[2] • What are the implications of these developments? • Why do you think these are important?	Understanding people’s perceptions of the situation and identifying possible changes in policy and practice Getting an idea of the way in which the issues in question are perceived by interviewees, and get a range of various perspectives/understandings of the developments, causalities and people’s values in relation to issues. QAF: Rel1, Rel2, Rel3
5. Who are the key players in the discussion, policy, or practice of young people’s engagement in post-conflict truth-telling/transitional justice processes?	• What role do government/academic/NGO/international/private sector/communities play[3]? • In what ways have they (each) been influential?	Understanding people’s perceptions of who is who in changing policy and practice Getting an overview of who people consider as key actors in the process. This question will also provide insights about the power dynamics between the stakeholders (e.g. who’s got power over whom). QAF: Rel1, Rel3
6. What information/knowledge has been the most influential in related to young people’s engagement in post-conflict truth-telling/transitional justice processes?	• Who is promoting the information/knowledge or event in question? • In your opinion, has the information [what they mentioned] influenced policy and practice? How? Probe for examples	Understanding what kind of knowledge is used in decision-making in general Getting a better picture of what kind of knowledge and other factors are influencing [topic], and from where the ideas are coming. More detailed information about possible changes in policy and practice because of new information/scientific knowledge QAF: Rel1, Rel2, Rel3

Table A2.

Understanding links between knowledge sharing and decision-making processes (purpose to assess important sources of influence on policy and practice)

Main question	Probes		Intent: what we are trying to find out Do NOT ask these directly
7. When doing work related to young people’s engagement in post-conflict truth-telling/transitional justice processes, where do you (or your organization) get the information you need to do your work?	• What kinds of information? • How does that information help guide decisions around what your organization does?		Understanding what kind of knowledge is used in decision-making in general Getting a better picture of what kind of information is seen as important and/or used in decision-making (scientific or non-scientific). QAF: Rel7, Eff2
8. Do you use scientific information in your work in relation to young people’s engagement in post-conflict truth-telling/transitional justice processes?	• How has it influenced or contributed to your work? • Where did you get that information? (Any specific events, publication, meetings, etc.) • What are the main barriers to using scientific information?		Understanding what the role of science is in decision-making Getting a better picture of the ways in which scientific knowledge is used by organisations, how they get the science they use, and what prevents them from basing their decision-making on scientific research findings QAF: Rel7, Eff2, Eff3
9. Which factors are influence your (personal and/or organization) decision-making around issues related to young people’s engagement in post-conflict truth-telling/transitional justice processes?	• Political factors • Individual or organizational advocates • Scientific information/ research • Are there any additional factors?	• Political factors • Public opinion • Precedent in other jurisdictions • Global pressures/ influences	Understanding what other aspects influence decision-makingUnderstanding how people see decision-making situations, which aspects matter most in making changes in policy and practice, and how research findings matter in relation to other factors

Table A3.

Determine respondent’s awareness of and/or involvement in the researcher’s project

Main question	Probes	Intent: what we are trying to find out Do NOT ask these directly
10. Have you heard about [researcher]’s research on young people’s engagement in post-conflict truth-telling/transitional justice processes? *if they do not recognize the researcher’s name, prompt with details about the project	[to non-partners] • What do you know about the research project? • How did you hear about it? • How would you describe your interactions with the project or the researcher? (e.g. presentations, workshops, etc.) [to partners] • How did you get involved in the project? • What was your role in the project? • What was your contribution to the project? (e.g. meetings, provide information, connect people, make recommendations, etc.) • Do you think that your input was taken into account?	Understanding awareness, role, and length of engagement with relevant actors and/or project partners Finding out informant’s awareness and opinions about the project. Finding out to what extent the degree and length of engagement in the project may be associated with changes in policy and practice. QAF: Rel3, Rel7, Cre7, Cre8, Leg1, Leg2, Leg3, Leg4, Eff2
[Ask 11 ONLY to participants and those who said they know the researcher and the project]
11. How would you describe your participation/collaboration experience in the project?	• How would you characterize your opportunity to participate and engage in the research? (i.e. rigid/restricted by student, open/facilitated by researcher/ participatory) • Do you have any suggestions regarding how engagement/participation could have been made more meaningful for you? • Do you think any key stakeholders were excluded from the research? • Any examples of positive experiences/what was done well? Any promising practices? • How could the participation/collaboration work even better in the future?	Understanding personal experience and feedback Further details of the influence of the project on the personal level, possible additional aspects (re: knowledge translation). Potential for improvement. QAF: Leg2, Legt3

Table A4.

Perceptions on design and implementation elements and how the programming at Royal Roads University supports student success (ask only to members of the research advisory committee)

Main question	Probes	Intent: what we are trying to find out Do NOT ask these directly
12. How do you think the [RRU program] program helps to support effective student research?	• How is research taught in the program? • How is the applied research focus reflected in the program? • How do you think [the researcher]’s project was influenced by the program (positively, negatively)?	Understanding program influence on effective research practice QAF: Cre1, Cre5, Cre6, Cre8
13. How was [the researcher]’s project assessed?	• What criteria were used? • What would you say are some of the challenges of assessing research of this kind?	Understanding how student research is assessed, and how advisory committee members conceptualizes research effectiveness
14. How would you characterize the design and implementation of [the researcher]’s project?	• Did [the researcher] demonstrate a comprehensive understanding of the context and elements relevant to the research problem? • How would you describe the application of the methods? • Was the execution suitable to the research objectives? • Was the execution suitable to the context? • Do you think resources were sufficiently and effectively allocated? • Were there any issues with the design that you can recall? How were these addressed? • Do you think any important stakeholders were excluded?	Perspectives about project design and implementation QAF: Rel3, Rel5, Rel6, Cre1, Cre4, Cre7, Cre8

Table A5.

Research outcomes assessment (ask only if they are aware of the project) (purpose to determine extent of outcome realization and research influence on knowledge or social process contributions around [topic])

Main question	Probes	Intent: what we are trying to find out Do NOT ask these directly
15. What contributions do you think [the researcher]’s project has made to young people’s engagement in post-conflict truth telling/transitional justice processes?	• Changes in knowledge/understanding? • Changes in attitudes? • Changes in skills? • Changes in relationships? • Changes in behaviour? • At what level do these changes mostly occur? (i.e. organizational, individual, governmental, policy, practice) • When did these changes occur? (during, post-project) • What are the implications of these changes? • Were there any negative outcomes of this project? If yes, please describe. • Probe for specific outcomes the researcher thought the informant could speak to. • What do you think the researcher did well to realize these results? • How accessible did you find the results and communication during the process? • Do you think the research can be transferred to other contexts?	Understanding the respondent’s opinion about the contributions of the research. Finding out the respondent’s opinion on the student’s research contributions (without leading to specific outcomes). Can give an indication of the utility of the research. Finding out how the student’s research is/was perceived and conceptualized by interviewees to get an overall characterization of the change process. This will help us construct narratives about alternative and/or supplementary theories of change. Finding out about the explicit outcomes/impacts of the project in question anywhere (in the world) of which the informant is aware, not just within their own work/organization. QAF: Rel6, Rel7, Cre7, Cre8, Cre10, Leg3, Eff1, Eff2, Eff3, Eff4
16. Has the research contributed to or influenced your work on the topic?	• What were the most important things you learned? • Have there been any positive or negative impacts on knowledge, awareness, policy, capacity, or practice? • In what ways? [ask for examples] • [If respondent mentions knowledge, ask about what knowledge product it came from]	Understanding how the student’s research has influenced their work (re: the topic of focus). Finding out about linkages between project and informant’s work on the topic of focus*, and whether the research has contributed to changes in policy and practice, the debate, awareness in the topic, knowledge, capacity, or any other type of contributions. Getting a sense whether the change is perceived as positive or negative. QAF: Rel5, Eff1, Eff2, Eff3, Eff4
17. If there was more time and resources available, what do you think [the researcher] could have done differently to produce more useful findings and/or change?	• Why do you think these would be useful? [ask for examples] • How do you think [the researcher] could have integrated these into their project? • Why do you think this [suggestion] was not done? • Do you think resources were efficiently and appropriately allocated?	Understanding alternative ToCs and perspectives of the research potential beyond what it did realize/intended to, and other opportunities. Hold to the end of the interview – if the interviewee starts talking about it at the beginning, please lead them back to any of the questions above and ask to return to the question. This Q allows participants to give feedback to the project and helps identify gaps/challenges, but we know many of the problems already and do not want to let this dominate/ mislead the main focus of the interview. Use this opportunity to increase the depth of any previous answers by probing and relating this question to any other points informants raise – if/when appropriate. QAF: Rel3, Rel5, Rel5, Rel7, Cre1, Leg3
18. What would have happened in young people’s engagement in post-conflict truth telling/transitional justice processes if this research had not been conducted?	• Probe to clarify if needed (the role of the project in improving collaboration, social networks, participation, engagement, etc.)	Testing “zero hypothesis”. Using a different angle to understand the true influence of ICRAF by asking what would be different had ICRAF not done its work. QAF: Eff4

Table A6.

Closing Questions

Main question	Probes	Intent: what we are trying to find out Do NOT ask these directly
19. What does effective research mean to you?	• What does effective research look like?	Understanding opinions on research effectiveness.
20. Do you have any additional remarks with regard to the role of [the researcher]’s project, or research in general, in change processes?	• Is there anything else you would like to add that has not been discussed that will be useful for our evaluation?	Closing Last remarks, things they might want to add that were not addressed, and closure

Table A7.

Summary of the TTP outcome assessment, with supporting evidence and consideration of contextual factors and causal mechanisms affecting outcome realization (Claus et al., 2020)

Results	Illustrative evidence
Outcome assessment	Summary of supporting evidence for the assessment	Contextual factors and causal mechanisms affecting how the outcome was realized
Current and prospective RRU students learn from PI’s research experiences Realized, clear project contribution [intermediate outcome]	Documents: Reflections from the PI’s research experiences are documented in a book that compiles experiences from the first DSocSci cohort. Interviews: The PI currently teaches research methods at RRU and is frequently invited by colleagues to share their doctoral experiences with students	Research skills taught and acquired at RRU are aligned with the skills expected of their faculty. The doctoral project helped expand the PI’s research experience, and developed relationships with colleagues at the university. The PI had prior research and practical experience in the field that also contributes to the way in which these experiences are presented to students. Accreditation acquired through the DSocSci program was a positive factor in the PI’s career trajectory
PI has increased opportunities to share insights and guide practice Realized, clear project contribution [intermediate outcome]	Interviews: The PI was invited to participate in the development of the Kampala Recommendations, the Sri Lankan truth commission panel, and academic webinars, where TTP knowledge and expertise were transferred. For example, the Sri Lankan conference clearly established the importance of the role of children in TJ so that every TJ draft law on truth and reparations have a clear mention of the Rights of the Child	The PI generated and acquired opportunities both through their approach (strong focus on building relationships) to the research process and their professional experience prior to the research. The PI was successful at creating and capitalizing on opportunities to share insights and guide practice through active engagement in the topic of youth engagement prior to, during, and after the project. They have been awarded these opportunities in part due to the doctorate accreditation acquired through completing the project. The Sri Lankan conference was attended by many experts, and changes reflecting the role of children in TJ were a collective effort
Trust and relationships built between research team and participants Realized, clear project contribution [intermediate outcome]	Post-project Survey: The majority of participants noted they felt safe to share confidentially, and that they could seek assistance if needed. Participants described the research team as “respectful”, “humble”, “polite”, “jolly”, and “listen well”. Several participants commented that the “researchers made me happy”. Interviews: Members of the research team characterized the working relationship between the team and the participants as “family-like”	The PI facilitated the process for trust-building by dedicating time and resources to training research assistants on participatory research and ethics. The nature of PAR activities and time set aside during the process built trust with participants. Participants perceived the research team to be respectful, indicating that the processes were carried out with due care for participants’ wellbeing
Partners’ priorities change Partially realized, clear project contribution [intermediate outcome]	Interviews: One partner noted that the work influenced the approach to their work, they thought more about young people’s engagement in TJ, and have taken up some initiatives to address it as a result (e.g. organizing the Kitgum Festival, TJ curriculum review for secondary schools). Other partners did not comment on the specifics of how their priorities changed, but did note the emphasis on youth in their community-led initiatives	Priorities and approaches to working are constantly in flux to respond to various factors including donor interests, research, and community demands and perspectives. Partners were strategically selected by the project as organizations that would have the potential to value and recognize the benefit of meaningfully and ethically engaging young people affected by conflict to support their overall mission or vision
Changed attitudes of research team around value of young people’s voice and engagement Realized, clear project contribution [intermediate outcome]	Interviews: Members of the research team noted their involvement in TTP improved their understanding of working with young people that they did not have before (e.g. active listening, ethical considerations of working with young people, the dynamics and the importance of working with young people) that increased the perceived value around young people’s voice and engagement. Indicator: All research team members continue to work with young people in their professional careers in some capacity	The research team went through a competitive hiring process, and part of the assessment for recruitment was the interest in working with young people. Candidates selected on that basis would have had a greater propensity for recognizing the value of young people’s voices and engagement, and perhaps a predisposition to advocacy for young people. Partway through the project, one of the research assistants left and was replaced
Research assistants gain new skills, professional exposure and build professional networks Realized, clear project contribution [intermediate outcome]	Interviews: All research team members noted that the experience with the project was positive for them in terms of acquiring new approaches and skills (e.g. active listening, ethical considerations of working with young people, the dynamics and the importance of working with young people) to work with young people through their active participation in the research design and implementation. One team member participated in an international conference and publication following the project which added to their professional experience and networks on the topic	The PI established a team of local researchers and allocated time to train them in participatory research methods and ethics. The PI treated team members as equal partners in the research, always seeking their validation and input to the design. This created a sense of ownership and pride over the work. Participation in the project involved new professional exposure and relationship-building with partner organizations. The research assistants also came to the project with enthusiasm, prior research experience, knowledge of the Ugandan context, and local networks
Research assistants have enhanced career opportunities in the transitional justice sector and work with young people Realized, clear project contribution [EoP outcome]	Interviews: One research team member co-presented the findings at an international conference with the PI. By working closely with the partner organizations during the project, one member of the research team was hired by one of the partner organizations. Indicator: Following the project, all members of the research team continued working with young people in issues related to TJ	The research project offered research assistants interested in working with young people and TJ the opportunity to gain experience. Research assistants had pre-existing interests in the topic. Partway through the project, one of the research assistants left and was replaced
RLP prioritizes young people in their work Realized, clear project contribution [EoP outcome]	Interviews: Informants noted that the research influenced RLP’s approach to their work. After the project, they thought more about young people’s engagement in TJ and have taken up some initiatives to address it as a result, including a collaboration to pilot secondary school curriculum that integrates TJ. Website: RLP empowers young refugees through support groups as a way of enabling them to build the social support network they need to enhance their mental and psychosocial resilience to cope with and overcome their distress in exile	Organizations’ priorities and approaches to their work are constantly in flux in response to various factors including donor interests, research on a given topic, and community demands and perspectives. RLP’s mission is to empower asylum seekers, refugees, deportees, internally displaced peoples, and host communities to enjoy their human rights and lead dignified lives; RLP was strategically selected as a partner because they had the potential to value and recognize the benefit of meaningfully and ethically engaging young people affected by conflict to support their overall mission or vision
RLP apply creative methods in their work and share within their networks Realized, clear project contribution [EoP outcome]	Interviews: RLP currently applies creative methods and approaches that emulate the TTP in their work with young people. For example, drama is used to demonstrate refugee youth vulnerability to mental health issues. RLP has shared the methods through their participation in the Child Protection Working Group, the TJ working group, other CSO meeting platforms, and other TJ fora in Africa	The project demonstrated the utility and versatility of creative methods when engaging young people to its partners. RLP has a member on staff that previously held a position that involved sharing and promoting creative methods to other organizations working with young people and TJ
Participating young people develop social and communication skills Realized, clear project contribution [EoP outcome]	Post-project Survey: Participants appreciated the research activities and approach, and many commented about their ability to share their stories and talk to people after having participated in the project. Interviews: Both researcher and practitioner informants believed the main contribution of the project related to the benefits gained by young people participating in the research process	The project provided young people with a unique opportunity to share their personal history that was highly relevant and meaningful to them. Participation using creative methods gave young people the opportunity to develop communication and social skills to express themselves on their own terms. Young people affected by conflict face psychological issues as a result of trauma that affects how they communicate and interact socially. In African culture, children are typically expected to be seen and not heard
Participating young people develop relationships with other participants Realized, clear project contribution [EoP outcome]	Post-project Survey: Participating young peoples’ responses to the survey indicated appreciation of the opportunity to share and meet other young people. Interviews: Members of the research team commented on the relational aspect of the research in that it supported a feeling of unity amongst participants, developed social networks and relationships, and was a positive experience for participants	The project’s engagement activities by their design and intent created space for relationships to develop, as participants were brought together in group settings on frequent occasions with an explicit objective to develop a sense of community in a safe environment
Participating young people have confidence to share with adults Realized, clear project contribution [EoP outcome]	Survey: Participation in the research activities boosted participants’ confidence to speak the truth with adults and with their peers. After participating in the research activities, respondents described a new sense of courage and freedom	A goal of the project was for young people to recognize the value of their voice and agency, which was built into activities and discussions with participants. Through this recognition, participating young people gained confidence to share their ideas and thoughts with the project team and other participants throughout the research process. Opportunities to share in a safe space through the project helped develop participants’ confidence to share with adults outside the project. TTP was facilitated in an inclusive, respectful, and sensitive way that was open to listening and hearing what and how young people wanted to share. In African culture, children are typically to be seen and not heard
Participating young people gain knowledge of truth commissions/transitional justice Realized, clear project contribution [EoP outcome]	Post-project Survey: Participants learned about other countries’ truth-telling processes, about what happened in the Ugandan conflict, where to go for support, and that they have the right to share their stories on their terms. Interviews: One of the project objectives aimed to increase participants’ understanding of TJ, as many participants were not aware prior to the research	The project actively sought to improve participants’ knowledge of truth commissions and TJ processes. RLP is currently involved in a secondary school curriculum review at the national level in collaboration with the Ministry of Education to integrate TJ into the curriculum to expand the knowledge of truth-telling commissions and TJ processes to all young Ugandans
Participating young people recognize the value of their voice and agency in truth commissions/transitional justice processes Realized, clear project contribution [EoP outcome]	Post-project Survey: Participating young people reflected after the research process that they believed everyone should be given the opportunity to share their experiences and they felt that opportunities for (participation in) TJ now exist for them	The recognition of participating young people’s voices and agency in truth commissions was encouraged through active listening and facilitation of culturally sensitive, respectful, and inclusive activities by the TTP to reinforce the importance of their voices. The activities facilitated meaningful and safe processes to collect individual stories and accounts of their experiences. In African culture, children are typically expected to be seen and not heard
Young people are consulted by RLP on issues that affect them Realized, clear project contribution [EoP outcome]	Documents: RLP facilitated a workshop with government representatives, NGOs, academics, lawyers, and young people; the overall objective was to discuss and refine the draft TJ policy for Uganda. Interviews: RLP has continued to work with young people and think critically about how to do it in meaningful and ethical ways. Website: RLP has consulted young people on a wide array of issues including: TJ, mental health, peer pressure, sexual relationships, peace, accessibility to social services, and early marriage through their Media for Social Change Program	The project’s strategic partnership with RLP demonstrated the value of and principles to apply when working with young people, which contributed to a shift in priority for the organization to engage young people. However, donor interests also influence how RLP functions as an organization as they are accountable to the funds they receive
National actors learn benefits of and how to engage young people Partially realized, insufficient evidence for project contribution [intermediate outcome]	Interviews: The project was able to demonstrate the value of engaging young people regardless of the political situation by engaging with young people in a tense political climate on a sensitive issue. RLP led a line ministry training of government officials through which they channeled the research results in October 2013. Uganda’s national TJ policy was reviewed by 20 young people who participated in the project in a process led by CSOs including RLP and ICTJ. Documents: Uganda’s national TJ policy highlights some gaps to guide future policy (e.g. to develop policy on children born while mothers were in captivity of the armed groups). It is unclear the extent to which other national actors beyond those close to the project understand fully how to engage young people in this context, as there are no specific guidelines outlined in the transitional justice policy. Indicator: In June 2019, the Ugandan government committed to and approved a national TJ policy which includes the best interests of the child as a guiding principle, indicating a formal recognition of the benefit and value of engaging young people in national TJ processes	TJ processes external to the project are ongoing in Uganda; for example, a TJ policy was under development and training workshops had been held with national actors. Relevant national actors present at a line ministry training workshop facilitated by a TTP partner organization to review Uganda’s TJ policy would have likely learned of the benefits of and how to engage young people through participating. The cultural stance in Uganda on young people is typically to be seen, not heard, a perspective prevalent across African culture
Young people are involved in a dialogue to review a draft national policy on truth commissions/transitional justice Realized, clear project contribution [EoP outcome]	Interviews: Partners discussed how the multi-stakeholder workshop and policy dialogue (which included young people who participated in TTP) led to the development of Uganda’s national TJ policy. Documents: Uganda’s national TJ policy was approved by the Ugandan cabinet in June 2019	The policy dialogue was facilitated by one of the project partners. The dialogue involved some of the members of the young people’s advisory who participated in TTP and some members of the research team. The Justice Law and Order Sector struck a committee in 2007 to develop a national TJ policy for Uganda and was in the process of consultations to develop the framework during the project, indicating TJ policy was on the political agenda

Table A8.

Summary of the PDAP outcome assessment, with supporting evidence and consideration of contextual factors and causal mechanisms affecting outcome realization

Results	Illustrative evidence
Outcome assessment	Summary of supporting evidence for the assessment	Contextual factors and causal mechanisms affecting how the outcome was realized
Other researchers become aware of the research Realized, clear project contribution	Documents: The proposal outlines the PI’s intention and strategies to reach academic audiences; presenting at conferences was an influential avenue to network and share findings with other researchers, and the feedback received from interdisciplinary researchers was positive; the SSHRC Storyteller Showcase required a short video summarizing the research to facilitate further awareness-building; the PI’s chapter contribution to the DSocSci book and promotion of the book could have spread awareness of the research. Web media: The dissertation is available online, via the RRU library repository for research and the PI’s website, making the research easily accessible; this website also hosts a blog, which shares reflections on the research experience and findings. Interviews: Researchers within the PI’s sphere of influence were aware of the project context and its findings. Indicators: The PI attended and presented at five conferences targeted to academic audiences; once available online, the PI received an immediate request to use the findings; being an award winner of the SSHRC Top 25 Storytellers Award and IISS Graduate Scholar Award likely drew academic attention to the research; altmetrics of book chapter downloads [132] indicate awareness.	The dearth of academic literature on the research problem presented entry points for the project and opportunities to bring awareness to academic circles. Attendance at and participation in conferences increased the PI’s capacity to reach and influence academic audiences. The prestige that accompanies research awards likely drew additional attention to the research. The availability of the dissertation online facilitates researcher access. Contribution to the DSocSci book was opportunistic, but the PI’s efforts successfully leveraged the opportunity for further knowledge mobilization. Diversification of knowledge products and dissemination strategies appears to have facilitated researcher awareness. The PI did not fully leverage knowledge mobilization through academic peer-reviewed publications. Access to RRU library resources following graduation posed a significant barrier to publish. Other researchers who may find the research valuable may not be aware of the findings owing to inappropriate search terms.
RRU and UVic students learn from the PI’s research experiences Partially realized, clear project contribution	Interviews: Researchers had positive impressions of the guest lectures given by the PI, noting that learning objectives were fulfilled (e.g. shared international fieldwork experience, shared experience conducting qualitative research, gave insight on the development sector, etc.) and gaps addressed (e.g. provided examples of applied qualitative research, provided examples from the African context, etc.). Web media: The Centre for Global Studies’ website outlines the expectations of visiting fellows, which include teaching; the website contains the PI’s profile. Indicators: Invitations to guest lecture; invitation to join UVic’s Centre for Global Studies as a visiting research fellow.	The PI was invited to guest lecture by professors within their network, established either from a personal or working relationship. These invitations are indicative of the PI’s expertise. Informants based their impressions on course evaluations or conversations with former students; the former is not a reliable indicator owing to low response rates. The extent of guest lecture effects and subsequent changes in behaviour are unclear; for example, student interest in the international field may be inherent to the student or inspired by the guest lecture, professor, or other factors.
The PI supports Moshi organizations participating in the research Realized, clear project contribution	Documents: Activities (e.g. sharing findings, offering pro bono services) to support participants were built into the project design. Interviews: The PI shared findings at two stages of the project; former participants appreciated the second round of interviews where preliminary findings were shared, discussed, and validated; researchers commended this research practice; all former participants interviewed noted they had received the results from the PI at project end; external to the project, the PI supported Moshi organizations via pro bono consulting (e.g. organized focus groups, facilitated strategic planning sessions, other). Indicators: Other individuals and organizations approached the PI to participate in order to get access to the research results.	Supporting organizations in Moshi was both a personal and project objective. Intentional planning, adequate funding, PI’s commitment, and implementation of activities supported the fulfillment of this objective. The PI’s communication of planned reciprocity to potential participants was clear and attractive to organizations. Unexpectedly, information about the PI’s planned reciprocity spread through participants’ networks to attract and incentivize participation to other individuals and organizations working in PDA in Moshi. Organizations in Moshi were both interested and open to improving their practices.
The PI built trust with participating organizations Realized, clear project contribution	Documents: Building and maintaining trust was reflected in project documentation and the dissertation; commitment to maintain anonymity was explicitly documented. Interviews: A couple informants reflected on the PI’s commitment to anonymity as a factor of trust in the project; most former participants reflected positively on how the PI shared findings (during second round of interviews, at project end); one researcher believed the validation and feedback approach used in the second round of interviews supported trust, agency, and ownership over the findings; former participants appreciated the discussion-like interview style, which made the PI come across as personable, relatable, and trustworthy. Indicators: Individuals and organizations were willing to participate in the project; evaluation informants were willing to participate in the evaluation at the PI’s request.	The PI planned and implemented strategies to support trust-building with participants. The personality, openness, and professionalism of the PI supported inter-personal trust and trust in the project. Organizations’ openness and willingness to participate in the project enabled trust to form (applicable to both ‘thriving’ and ‘floundering’ organizations).
The PI built relationships with participating organizations Realized, clear project contribution	Documents: The PI was aware of being an outsider and the importance of building relationships; PI’s intentions for anonymity, the interview approach, extended fieldwork periods, and periodic communication were explicitly documented. Interviews: Former participants and researchers discussed how the interview approach fostered trust and connection between the PI and participants (e.g. collected data valuable to participants, in-person interview discussions); offer of pro bono services as a form of professional relationship-building; PI visited the community multiple times and spent over ten months in Moshi; PI maintained periodic communication with participants during, at project end, and post-project; former participants described the PI as personable and relatable. Indicators: Strategies for trust-building (e.g. protecting participant anonymity, sharing findings) align with relationship-building strategies.	Being an outside researcher and other aspects of positionality could have acted as barriers to the PI’s relationship-building and overall project; however, neither appears to have played a significant role. The PI’s personality and nature were facilitating factors, though it is evident the PI also put effort into building relationships (interview approach, pro bono services, communication, maintaining anonymity, sharing findings). Some participants got connected with the PI through mutual friends or other pre-existing relationships.
Participating organizations change approach based on preliminary findings/ lessons from the research Partially realized, clear project contribution	Interviews: The findings predominantly prompted reflection on current organizational practices for former participants; actions by participating organizations in response to the findings include: reassessment of practices, a change in project proposal or reporting protocol (added question about unintended consequences of project activities), and planning for the future leadership of the organization; one former participant was unsure if changes had taken place in the organization, but had themselves taken up the findings and used them when they moved to a different organization; for most former participant informants, the findings affirmed the direction of their organization and the effectiveness of current approaches; some informants found the findings valuable but did not believe any major changes in their organization were catalyzed as a result; three former participants were unsure if practices had changed; two former participants noted that the findings had no direct influence on their organizations’ practices.	The structure of the research process (conversational interviews, follow-up interviews to share preliminary findings), the lessons presented in the findings, and the PI’s ongoing contact and relationships with project participants all contributed to changes in awareness, reflective practices, and new actions taken by participating organizations. Whether informants interviewed were from ‘thriving’ or ‘floundering’ organizations would likely influence the degree or types of changes necessary for or observable within their organization. For example, ‘thriving’ organizations would logically have little to change or have a pre-existing orientation toward reflexivity; hence, the predominance of responses indicating the findings affirmed organizations’ current practices. Informants willing to participate in the evaluation may also predominantly come from ‘thriving’ organizations, though this information was not disclosed to the evaluators. Changes that result from conversations and reflections on practice happen naturally within and between organizations.
The PI has enhanced personal knowledge/ insights on PDA and proposal writing skills to apply in consulting work Realized, clear project contribution	Documents: The PI’s intention to apply learning and insights into their professional consulting was documented in their proposal and dissertation. Interviews: Most practitioner informants and some researchers had the impression that the PI shares insights from their doctoral experience via their consulting with organizations and conferences; practitioners shared their awareness or evidence of how the PI’s insight-enhanced consulting has helped six PDA organizations in Tanzania, Canada, the United States, and beyond; informants identified a variety of skills and enhanced insights that the PI now brings to their consulting work, including facilitation skills, diverse PDA perspectives, academic and analytical thinking, organizational governance and management, leadership, monitoring and evaluation, and cultural and contextual awareness. Two informants described the PI as a valuable asset to their organization’s professional network.	Gaining applicable insight and skills were explicit goals for the PI in terms of their professional growth since they began their doctoral experience. Most practitioners and some researchers linked these gains directly to the doctoral process and experience. The PI’s role as a professional consultant to PDA organizations enables the PI to apply and transfer their personal knowledge and enhanced skills to their consulting and the organizations they work with and through. The PI identified the openness of participating organizations as an important factor that supported the enhancement of their knowledge and insights on PDA. The PI’s pre-doctoral consulting experiences and insights may have already equipped the PI with enhanced PDA knowledge, which may have been complemented by the PI’s doctoral research. The PI has an intrinsic orientation to continuous and lifelong learning, such that they will build on and apply their knowledge from every experience they have.
The PI mentors other DSocSci students Realized, indirect project contribution	Interviews: Informants attested to the opportunity for the PI to mentor another DSocSci candidate through RRU; the mentorship included application, coursework, and research committee advice; the PI and the mentee met periodically during their programs, and fostered a deeper friendship which evolved into a professional working partnership at the mentee’s organization; one researcher felt that the PI made a positive contribution to the DSocSci program as a whole.	The opportunity to mentor another DSocSci student was by chance, but fostered a serendipitous professional partnership between the mentor and mentee. Factors that facilitated this relationship included similar interests and backgrounds in development, working in Africa, and sharing the same supervisor.
Transform International collaboration Realized, clear project contribution	Interviews: Several informants were aware of the PI’s involvement with TI; practitioners from TI commented on the teamwork between the PI and other TI staff, the complementarity of skills each member brings to the collaboration, and highlighted the PI’s knowledge on organizational management; several practitioner and researcher informants discussed how the PI has applied their knowledge and approach from their doctoral research into the TI collaboration.	Collaborating with TI was an unexpected outcome. The TI collaboration emerged from the PI’s relationship with the DSocSci mentee, where upon connecting over their shared interests and professional backgrounds uncovered further parallels between each of their respective doctoral research projects and TI’s objectives. It is evident that the PI shares many of the same values as other TI staff, including passion about development, appreciation of learning, and valuing of each others’ strengths. TI is a collaboration-oriented organization owing to its mission and operating structure.
The PI gains professional capacity and recognition as PDA expert Realized, clear project contribution	Interviews: Informants described the PI as a ‘bridge’ between scholars and practitioners who is able to apply both their academic and professional expertise to their work; a few practitioners view the PI as an effective partner who brings valuable perspectives, skills, and contributions to their working partnership; one practitioner felt their organization had an ‘edge’ over other organizations because of the PI; one practitioner would collaborate with the PI on any project; several practitioners and researchers had the impression that the PI expanded their expertise as a result of the research, and gained recognition amongst organizations in Moshi during the research process. Indicators: The accreditation of a doctorate is widely recognized as an indicator of expertise; the PI was awarded prestigious scholarships and awards for their research (e.g. Mitacs Fellowship, SSHRC Fellowship, Top 25 Finalist Storytellers Award, Graduate Scholar Award); the PI received invitations to contribute to a book chapter and professional magazine, facilitate conferences, become a visiting scholar at UVic, guest lecture at UVic and RRU; following the research, the PI has joined new collaborations in leadership positions (i.e. joined the board)	Acquiring the accreditation of a doctorate was one of the PI’s aims in pursuing their research. The PI also intended to expand their consulting expertise to include research, as the DSocSci experience would build their research capacities. Some informants believed that the PI was already a PDA expert prior to joining the DSocSci program, and had years of extensive professional experience as a practitioner

Table A9.

Summary of the SNDP outcome assessment, with supporting evidence and consideration of contextual factors and causal mechanisms affecting outcome realization

Results	Illustrative evidence
Outcome assessment	Summary of supporting evidence for the assessment	Contextual factors and causal mechanisms affecting how the outcome was realized
Informants’ participation in research prompts reflection on WASH Realized, clear project contribution [intermediate outcome]	Interviews: Practitioner participants noted thinking more deeply as a result of the SNDP on how to support the development of the WASH sector and how to handle negative sanitation practices in riverine communities by considering solutions that are not conventional. The PI suggested that the interview and opportunity to discuss personal experiences may have been affirming for research participants	The PI facilitated the process of reflection by involving stakeholders in data collection and asking them about their WASH knowledge and practices (co-production of knowledge). Engagement with the research questions and other system actors through the WhatsApp group exposed informants to different perspectives and ways of thinking
PIND staff gain knowledge and capacities and apply them in related work Realized, clear project contribution [intermediate outcome]	Interviews: PIND staff were aware of the SNDP which coincided with their own work. The SNDP presented new lenses to explore sanitation programing which have been adopted by PIND staff. The SNDP contributed to knowledge within PIND and has changed the way staff work on sanitation in riverine communities. However, it was suggested that PIND could have leveraged SNDP contributions further. Survey: Survey results suggest that the SNDP provided practitioners, including those outside of PIND with a better understanding of the topic. Some practitioners are using the SNDP’s findings to improve their role as a WASH community worker	The PI simultaneously conducted their doctoral research while working at PIND. Through the PI’s managerial position at PIND, knowledge gained by the PI was transferred to PIND colleagues also working in the WASH sector (n.b., PIND works in many areas of community development, WASH being only one focus). Learning from the PI as a colleague (e.g. mentor relationship) increased PIND staff effectiveness and capacities. As a result of staff enthusiasm and dedication to improve their skills as WASH practitioners, PIND staff were motivated to apply SNDP knowledge and capacities to their work
Forum for WASH discussion grows Realized, clear project contribution [end-of-project outcome]	Documents: The PI played a leading role in the establishment of the WASH sector WhatsApp group which brings together practitioners and different stakeholders working in the Niger Delta to share knowledge and best practices. Interviews: Practitioners are aware of the WhatsApp group and note that the PI continues to engage in the forum. Survey: The majority of survey respondents use the WhatsApp group more than once a week on average to connect with other WASH practitioners, use information shared in the group, share information to the group, ask questions to the group, and answer questions asked by other members in the group. Respondents also noted sharing job opportunities to the group and learning from best practices of other organizations who are successful in improving WASH, including PIND and United Purpose	Reflecting on their experience in the Niger Delta WASH sector, the PI identified the need for a mechanism for a community of practice for stakeholders and practitioners to discuss and engage with one another to share information. Prior to the SNDP, there was a lack of engagement and collaboration across the Niger Delta WASH sector which contributed to duplication and inefficient use of resources. By leveraging their networks and reputation within the WASH sector, the PI identified the benefit for practitioners and stakeholders to participate in the forum, which functioned as a community of practice or a coalition. Practitioners continue to engage in and use the forum, suggesting the WhatsApp continues to be a beneficial space for practitioners to participate
WASH sector recognizes WASH gaps, issues, and opportunities Realized, clear project contribution [end-of-project outcome]	Interviews: Practitioners connected contributions made by the SNDP supported the WASH sector in recognizing gaps, issues, and/or opportunities for the improvement of sanitation in the Niger Delta. The SNDP highlighted the need to explore appropriate technology options, the availability of materials within communities, and behaviour change. Practitioners have begun to prioritize the needs of individuals in riverine communities and new projects have stemmed from this recognition. The increase of information on the topic as well as the identified need to collate knowledge and research have supported practitioners’ abilities to identify gaps and ensure there is no duplication across projects. The SNDP provided a starting point for further work on the topic. Survey: Over half of survey respondents believe that the Nigerian WASH sector is now generally more aware of WASH gaps, issues, and/or opportunities than two years ago. However, the majority of survey respondents were unsure as to whether the SNDP had directly contributed to this increase in awareness	The SNDP actively sought to identify gaps in the Niger Delta WASH sector and was an exploratory study. By involving WASH sector actors and stakeholders in the SNDP and targeting findings and recommendations to local WASH practitioners and organizations, knowledge gaps were identified and filled. Practitioners are looking for information and direction to address WASH issues, and the targeted recommendations presented by the SNDP are useful, practical, and accessible to fill this knowledge gap and influence the agenda
Nigeria’s WASH sector has enhanced capacity for knowledge sharing and learning Realized, unclear project contribution [end-of-project outcome]	Interviews: The SNDP supported enhanced knowledge sharing and learning by identifying the need for improved coordination of activities in the sector to make significant progress. The SNDP contributed to outcomes of sector engagement (e.g. via the WhastApp group) leading to the increased collaboration between WASH sector actors to minimise duplication. However, some practitioners remain sceptical of knowledge sharing practices as some organizations are unaware of work that is underway in the sector. Survey: The majority of respondents agree that WASH practitioners in Nigeria share information with and learn more from each other now than they did two years ago. However, most were unsure as to whether the SNDP contributed to greater information sharing, with some respondents being unaware of the SNDP. One survey respondent suggested that there remains no strong synergy among WASH practitioners and other key players in the sector	Prior to the SNDP, the Nigerian WASH sector worked in silos. However, it is suggested that the need for collective engagement across the relatively small WASH sector was an issue that was known before the start of the SNDP. WASH sector actors recognise the benefit of a WASH-oriented community of practice and are open to participating in one. By highlighting gaps, disseminating and sharing findings to participants, and presenting engagement options (e.g. WhatsApp group), the SNDP enhanced local capacity for knowledge sharing and learning
WASH sector explores technologies in high water table contexts Partially realized, unclear project contribution [end-of-project outcome]	Interviews: Practitioners suggested that the SNDP triggered creativity around innovation involving appropriate technologies for the Niger Delta region. However, in some cases, it is unclear to what extent the PI’s work directly affected the initiation of these external projects. Informants noted that appropriate technologies have not yet been identified as the sector is young and naïve – this remains a work in progress. Survey: The majority of survey respondents were neutral or unsure of whether the WASH sector is exploring technologies in high water table contexts. There was also overall uncertainty regarding whether the SNDP had contributed to any changes in the WASH sector regarding the exploration of suitable technologies	By raising the issue that there are currently no appropriate technologies for the challenging geographical context of the Niger Delta, the SNDP identified the need for follow-up and further exploration into the topic. By disseminating this knowledge through appropriate methods, including leveraging the PI’s networks with local WASH organizations, other WASH practitioners in the PI’s sphere of influence became aware of this gap, subsequently influencing the research agenda
Local governments have enhanced capacity for improved policy and effective WASH practice Partially realized, unclear project contribution [end-of-project outcome]	Interviews: There is limited evidence for this outcome as government informants were not accessible for interviews. Six federally-funded local universities in the Niger Delta offer capacity-building courses for local governments. Survey: Partnerships between government and civil society have been built to intervene in the WASH sector. From federal to LGA leaderships, there is now a greater focus on ending open defecation. It is unclear the extent to which this initiative is connected to the SNDP. Documents: Efforts have been made by agencies such as UNICEF to train RUWASSA to build local governmental capacity. However, there remains no regional resource that government staff or community members can turn to for resource information, certification, or skills development	Government actors; involvement in the data collection phases of the SNDP likely stimulated learning, reflection, and skills-building. By highlighting gaps and presenting engagement options (e.g. WhatsApp group), the SNDP facilitated the enhanced capacity for knowledge sharing and learning with actors in the PI’s sphere of influence. External initiatives have been implemented to enhance government capacity (e.g. federally-funded university courses, UNICEF training); however, the Nigerian context and LGA priorities have proven challenging. Moreover, the lack of political will and commitment at the national level are major barriers affecting the improvement of WASH in Nigeria. As a result of poor funding and low capacities to implement WASH projects, local governmental agendas do not prioritise WASH
PI’s professional development enhanced by research experiences Realized, clear project contribution [intermediate outcome]	Interviews: Informants connected the PI’s credibility and heightened expertise in WASH to the SNDP. The PI is in an advisory role for WASH practitioners and organizations, and is a source of information on WASH issues. The PI’s professional and practitioner skills and networks were improved, deepened, broadened, and heightened by completing the doctoral research. The SNDP provided the PI with more motivation to discuss sanitation as an urgent issue and priority for organizations. Academically, the PI quickly became adept at research communication and developing a sustained argument. Survey: The PI is now a main source of information for the WASH sector. Documents: A doctorate is a life-changing experience which has permanently changed the PI’s depth of understanding, perspective, and approach to future work on the topic	The PI generated and acquired opportunities for professional development by leveraging and enhancing their reputation through professional networks (including PIND) and by involving key WASH stakeholders in the data collection process. Coalitions were created and strengthened through this process, which provided the PI with the opportunity to share their knowledge and research findings to other system actors. By being immersed in the doctoral research process, the PI acquired knowledge, skills, and understanding to assist their future work in the topic
The PI has deeper understanding of issues in WASH and how to approach working in challenging contexts Realized, clear project contribution [end-of-project outcome]	Interviews: WASH insights that are captured within the SNDP, as well as learning to work in developing contexts that have arisen out of the research process, have placed the PI in a position where they are able to provide useful ideas and support to collaborations in sanitation in terms of uncovering challenges, filling gaps and designing solutions. The research process provided the PI with an opportunity to explore topical issues that they were passionate about and of value to their future work. Documents: The build-up of knowledge and expertise from the research experience reinforced the PI’s understanding of the topic and has provided them with more tools when working in regions outside of the Niger Delta who face similar sanitation challenges. This has permanently changed the PI’s depth of understanding	The PI built their own capacity through the research process by using methods which required them to immerse themselves in the research experience. The knowledge gained through the research process and findings, coupled with leveraging of the PI’s networks, have allowed the PI to continue working in contexts similar to those of the Niger Delta
Knowledge network of scholars and practitioners share development knowledge and facilitate knowledge-to-practice transformation (TI) Realized, clear project contribution [end-of-project outcome]	Interviews: The PI is now the executive director of Transform International (TI), which has a primary focus on WASH. Informants noted TI as one of the most substantial outcomes of the SNDP. TI works with a network of similar NGOs across developing countries to share best practices and sustainability knowledge. The PI turned their doctoral knowledge into action and continues to share their knowledge through TI with other practitioners working in different developing countries (e.g. Malawi, Papua New Guinea, Kenya, Tanzania, and Bolivia) through a two-way knowledge exchange of best practices. Without the SNDP experience, it is unlikely that TI and its associated networks would have been formed. TI is noted to connect and align organizations working on a topic to improve efficiency and remove duplication of work. Documents: The PI is now the executive director of TI	By identifying the need for a knowledge sharing mechanism for the Niger Delta and other regions, the PI established an organization to share best practices and build-capacity of NGOs working in WASH. TI co-produces knowledge with practitioners who are experts across a multitude of fields to build capacity of local actors and facilitate knowledge-to-practice. This also supports coalition strengthening. By aligning TI’s work with the work of other NGOs, the PI contributes to sustainability in international development projects

Table A10.

Individual evaluator and average scoring of TTP for all QAF criteria, with justifications for the score allocated (Claus et al., 2020)

Principle	Criteria	E1	E2	E3	E4	Avg.	Justification/Comments
Relevance	Clearly defined socio-ecological context	2	2	2	2	2	Dissertation provides clear and full description of the country and problem context; reviews background information from multiple dimensions (historical, political, cultural, international regulatory standards/protocols, psychological, etc.); clear identification of entry points (literature gap, practice gap)
	Socially relevant research problem	2	2	2	2	2	Research problem is well-aligned with the Ugandan context; informants corroborate entry points and relevance/value of study; dissertation reflects consideration of the practical application of the research activities and outputs in multiple dimensions
	Engagement with problem context	2	2	2	2	2	Lead researcher had previous volunteer experience in Uganda, ten years of professional experience with IICRD, and experience with the Sierra Leone Truth and Reconciliation Commission (the first commission to systematically involve young people as active participants); hired local researchers which expanded influence potential; networked with local organizations working on the topic (RLP, TPO Uganda) which expanded influence potential; informants believed the research team engaged ethically and appropriately within the problem context
	Explicit theory of change	2	2	2	2	2	Dissertation explicitly identifies and documents the main intended outcomes, how they would be realized, and relevant progress markers
	Relevant research objective and design	2	2	2	2	2	Decisions on research design were guided by ‘do no harm’ and with the interests of young people at forefront; both objectives and design were co-generated within the research team; activities were piloted and revised based on feedback from YPRA participants; PAR-driven research design was relevant and appropriate to the problem context and objectives of the study to promote the authentic voice, needs, and perspectives of young people and build their capacities around safe and meaningful engagement in transitional justice processes
	Appropriate project implementation	2	2	2	2	2	Participatory execution was well-aligned to address potential problems within the problem context and satisfy the objectives; all activities were piloted with YPRA participants and revised based on feedback; diverse representation of young people affected by the conflict and partner organizations were engaged throughout the research process; informants viewed project engagement to be culturally sensitive and meaningful
	Effective communication	2	2	2	2	2	Research strategically identified and communicated with relevant actors (young people, partner organizations, local to national target audiences); strong internal dialogue processes between members of the research team; research activity communication was thought to be clear and appropriate; communication of research activities used multiple strategies to meet diverse needs (use of speech, writing, drawing, and both language and sign interpreters); verification workshops were held with participants to ensure understanding and accuracy of findings; planned a follow-up ‘evaluation’ workshop with participants post-project; dissertation is well communicated; produced diverse research outputs for multiple audiences (e.g. conferences, news articles, newsletters, articles, SSHRC story-telling, reports, etc.)
Credibility	Broad preparation	2	2	2	2	2	Literature review is extensive and comprehensive; external evaluator noted “This dissertation, which interweaves elements of transitional justice and truth-telling, psychosocial support, child protection, and research ethics, is an excellent piece of research that makes highly significant contributions to both knowledge and practice” (Doc6) and “The document makes appropriate and deep contact with the extant literature” (Doc6); dissertation expressly addresses challenge that PAR is not a theoretical framework, provides justification, and demonstrates true understanding of the approach
	Clear research problem definition	2	2	2	2	2	Research gaps are identified; research problem is clearly stated in the dissertation, grounded within the literature, and explicitly connected to the context
	Clear research question	2	2	2	2	2	Research questions clearly presented in the dissertation; transparency is given for the addition of a research question informed by participants; research questions are justified and grounded in the problem context
	Objectives stated and met	2	2	2	2	2	Objectives are clearly stated in the dissertation; objectives have been met
	Feasible research project	2	2	2	2	2	Project was supported by sufficient funding (SSHRC [Joseph-Armand Bombardier Canada Graduate Scholarship, Michael Smith Foreign Study Supplement], IDRC); research team and partner resources were thought to be appropriate; research activities/tools were flexible to adapt to different participants’ needs; some young people participants wished there had been more time for the activities; informants’ suggestions for what could have been done differently fell outside the bounds of a doctoral research project
	Adequate competencies	2	2	2	2	2	Research team demonstrated a diverse and complementary set of competencies to the project; lead researcher had extensive research experience; Ugandan members ensured cultural appropriateness and had relevant research experience; team sought support for language and signing resources where appropriate; lead researcher offered training to co-researchers and research assistants; even with research assistant turnover mid-project, the position was refilled
	Research approach fits purpose	2	2	2	2	2	Lead researcher brought in principled learning from DSocSci course, applying and reflecting multiple ways of knowing into the project design and outputs; utilized and adapted participatory methods/tools previously used at IICRD with input from partner organizations (RLP, TPO Uganda) and YPRA pilot; participatory approach of the research bridged the gap of young people’s voices in a transitional justice process which is normally regulated by adults; reflection on paradoxes/conflicts were brief, but present (re: inviting young people to share how they wished to share their experiences without coercion to actually share their stories); approach is justified in documentation and informants’ reflections demonstrate support for the approach taken considering the context and objectives of the research
	Appropriate method	2	2	2	2	2	Methods description is extensive and transparent in the dissertation; methods are fit to purpose; methods were tested with the YPRA pilot, representing the groups engaged in the main study; tools/methods selected were adaptable based on pilot participant and partner feedback/input, and reflect participatory principles central to the approach; activities were adapted to fit different participants’ needs and clearly justified
	Clearly presented argument	2	2	2	2	2	Results are clearly presented in the dissertation and throughout other research outputs; results presented in a logical manner, sharing young people’s perspectives by the different groups (BIC, FA, DBW, ID, etc.) – these were thought to be important findings to inform how to engage with different groups; clear demonstration of connection between evidence/results (excerpts and direct quotations), recommendations, and conclusions; alternative explanations are not explored (however, they do not apply to this type of research)
	Transferability and generalizability of the findings	2	2	2	2	2	Dissertation reflects on the extent of the generalizability of the findings for other contexts; informants believed the research process was highly transferable and an exemplary case of PAR; evidence of transfer of research approach to other contexts (e.g. RRU Resilience by Design Lab, lead researcher’s current research focus in cancer and healthcare context, participatory Ugandan curriculum development on post-conflict processes, external advisor’s community-based approach and inclusion of vulnerable voices, RRU’s School of Leadership Studies, supervisor’s work on the Colombian truth and reconciliation commission, supervisor’s working partnerships with First Nations in British Columbia); while findings are case-specific, many of the principles/recommendations are thought to be transferable by informants
	Limitations stated	2	2	2	2	2	Discusses shortcomings of the methods (e.g. PAR, sample size); discusses challenges/dilemmas encountered (e.g. research barriers, geography, cultural barriers, compensation to participants, managing expectations, delays); discussion on limitations of results are limited (e.g. representativeness), but there are indications from researcher interviews and internal project documents that limitations were accounted for on an on-going basis
	Ongoing reflexivity and monitoring	2	2	2	2	2	Inclusion of processes for reflection are explained (e.g. YPRA pilot, daily research team reflection [summary reflection form], journaling, post-project ‘evaluation’ workshop); some reflections and changes made based on those processes are presented transparently in the dissertation (e.g. added research question, adapted methodology, expanded inclusion of young people’s groups and ages, pursuit of new opportunities); summary reflection form well designed to include aspects for what the research team learned, what could be done differently, and what follow-up could look like
Legitimacy	Disclosure of perspective	2	2	2	2	2	Dissertation references funding sources, partners, and researcher positionality; dissertation discusses selection and justification for partnership and collaborations; dissertation discusses lead researcher and research team biases (gender, ethnicity, socio-economic background, education, privilege); dissertation discusses power dynamics and consideration for marginalization
	Effective collaboration	2	2	2	2	2	Internal project documents outline roles and responsibilities of the research team and partners; egalitarian decision-making processes within the research team; members of the research team reflected on the interpersonal relationships and collaboration experience positively; effective collaboration with partners and young people was central to the participatory nature of the project; lead researcher created genuine opportunities for other members of the research team to grow and gain professional experience (e.g. co-authoring of articles, presenting at conferences); partnership with RLP and TPO Uganda reflected on positively by research team and members of those organizations
	Genuine and explicit inclusion	2	2	2	2	2	Genuine engagement of young people’s voices and ideas into the project design (e.g. YPRA pilot) and results (participants); co-researchers/research assistants felt involved and invested in the project; co-researchers, partners, and supervisory committee felt their input was included throughout research process; values of participation, ethics, and inclusion clearly upheld throughout the research process; design reflected conscientious inclusion of gender dynamics, diverse groups’ needs, power dynamics, ‘do no harm’, and cultural appropriateness; built in opportunities to verify results with participants to ensure their voices were accurately represented; informants reflected on lead researcher’s social capital, orientation to teamwork, and sincerity
	Research is ethical	2	2	2	2	2	Project received ethical approval by RRU Research Ethics Board; project submitted proposal to the Ugandan National Council for Science and Technology, and received approval; approval also received from three local government districts in Uganda; research team practiced informed and ongoing consent with their participants throughout the research process; constant researcher reflection on principles of ‘do no harm’ and Right of the Child were at the forefront of decision-making and research practice; participants’ reflections on their engagement demonstrated the research was conducted ethically (e.g. “researchers were polite and good”, “confidentiality is kept”); follow-up verification of findings and ‘evaluation’ workshop were reflective of ethical research practice; practitioner perceptions indicate the research upheld the dignity of the participants
Effectiveness	Research builds social capacity	2	2	2	2	2	Participants’ reflections demonstrate learning occurred regarding the Ugandan conflict, transitional justice processes, how to engage/share with others, and sources of support/help; other researchers felt the research was valuable for the learning and opportunities given to participants; participants gained courage to share and express themselves, which they previously did not have; partners learned of actor gaps/opportunities in their work and how they could engage young people; practitioners believed the project made important knowledge contributions; co-researchers developed research skills, professional capacities, and networks; project influenced changes in perspectives/recognition amongst participants, the research team, partners, and practitioners (IICRD); informants spoke to positive perceptions of participants and partners they felt were influenced by the research personally and professionally; some practitioners were inspired by the project to reflect on or change how they approached their own work
	Contribution to knowledge	2	2	2	2	2	Research made a contribution to the knowledge base; built participants’ knowledge in the topic of transitional justice; built partners’ knowledge around persistent issues for their practice; some contribution made to the development to the Kampala Recommendations (some of the project’s findings are reflected in that document), though they have not been taken up or applied widely; shared findings through a webinar, an article in a special issue, relevant conferences, and the SSHRC Story-tellers competition to reach a wider audience than the dissertation; lead researcher made knowledge contributions to the Sri Lankan Truth and Reconciliation Commission; practitioners viewed research to make a significant contribution that was missing from the literature
	Practical application	2	2	2	2	2	Contributions shared and applied in Kampala Recommendations, partners’ practices (e.g. involvement of young people, Kitgum Festival, documentary), Ugandan Transitional Justice Policy, Ugandan curriculum development, Sri Lankan Truth and Reconciliation Commission; lead researcher and co-researchers continue to adapt and apply approaches in their work (e.g. RRU Resilience by Design Lab, RRU School of Leadership Studies, NMPDC, working with young people); members of advisory committee research influenced by project (e.g. Colombian Truth and Reconciliation Commission, reconciliation processes with First Nations in British Columbia)
	Significant outcome	2	2	2	2	2	Some participants felt the project changed their life; practitioner and researcher informants perceived the project supported and empowered the young people that participated; high potential that contributions supported positive social change through the Sri Lankan Truth and Reconciliation Commission; high potential that contributions to Ugandan Transitional Justice Policy will take effect; how co-researchers and partner organizations are engaging within the local context will have positive social implications

Table A11.

Individual evaluator and average scoring of PDAP for all QAF criteria, with justifications for the score allocated

Principle	Criteria	E1	E2	E3	E4	Avg.	Justification/Comments
Relevance	Clearly defined socio-ecological context	2	2	2	1	1.75	Proposal and dissertation provide clear and full description of the problem context from a global to local level (Moshi); dissertation outlines the Tanzanian private development aid context; clear identification of entry points (literature/knowledge gap); the decision-making context of the organizations investigated and how recipients perceive them were missing (justification for why the research is needed in the socio-ecological context)
	Socially relevant research problem	2	2	2	2	2	Research problem is well-aligned with the Tanzanian private aid context; informants reflect on relevance and value of study; dissertation identifies the value of the findings to diverse audiences (e.g. academics, policy-makers, non-profit practitioners, philanthropists/donors) to make better informed funding decisions
	Engagement with problem context	2	2	2	2	2	Researcher interacted substantially with the problem context (via personal experience, engagement with literature, and in fieldwork during the research process); researcher’s engagement was appropriate to objectives of the research; researcher was well positioned to define the research problem and influence the context (extensive international experience within the development and philanthropic-giving sector, interned during research with DAC, collaborations and consulting work aligned with research interests, access to extensive networks)
	Explicit theory of change	1	2	1	1	1.25	Dissertation identifies the main intended outcomes and how they would be realized, though brief and not actor-specific; supervisor notes that change was intended and built into the research (had a ToC, but this document was not shared); other informants remarked on their recognition of the intentionality behind the PI’s project
	Relevant research objective and design	2	2	1	2	1.75	Objectives are relevant and stakeholder needs/values are considered; decisions around research design are relevant, appropriate, and transparent; informants reflected positively on the design of the study; informants discussed the relevance of the research and its design
	Appropriate project implementation	2	2	2	2	2	Informants (participants, advisory committee, and practitioners) noted relevance of study’s focus on private aid in Moshi, Tanzania; extensive fieldwork supported deep engagement with the country and problem context; second round of interviews was regarded positively by participants and advisory committee; informants believed the array of organizations was appropriate, as was the extent of stakeholders representative of those organizations (e.g. end-use recipients, aid implementers, donors, partners, etc.); reflection and adaptation are noted transparently; adaptations made to project implementation to accommodate context
	Effective communication	1	2	1	2	1.5	Researcher planned and achieved extensive communication during (multiple interview sessions, frequent e-mail correspondence) and post-project (sharing findings with participants, dissemination and presentation of findings); researcher maintained contact with participants post-project (achieved more than typical); some informants noted communication on topic could have been clearer or done with more background/notice (one instance depends on a gatekeeper, however); multiple media were used to communicate the findings (dissertation, summary of findings, e-mails, conferences, practitioner magazine, book chapter, website/blog, SSHRC Storytelling video, etc.); several informants appreciated the clear and accessible writing style of the dissertation/outputs; some informants believed the PI should have written academic articles or a book targeted to practitioners
Credibility	Broad preparation	2	2	2	2	2	Researcher conducted comprehensive literature review (private development aid, philanthropy, grant-making, non-profit management, social venture/entrepreneurship, outcomes based and impact giving, development evaluation, etc.); dissertation demonstrates integration of disciplines and academic and grey literature
	Clear research problem definition	2	2	2	2	2	Research problem is stated in the dissertation, and value of understanding the problem from recipient perspectives is justified and grounded within the literature and Tanzanian context
	Clear research question	2	2	2	2	2	Research questions are clearly stated in the dissertation; justification and relevance of research questions to context are provided
	Objectives stated and met	1	1	1	1	1	Objectives are scattered throughout the research proposal and final dissertation, and are logically connected to the problem context identified; more explicit and clear documentation of objectives would have aided the structure and clarity of the project; objectives appear to have been met
	Feasible research project	2	2	2	1	1.75	Project supported by sufficient funding (SSHRC Fellowship, Mitacs [Accelerate Fellowship], IDRC), which enabled extended time for fieldwork and second round of interviews; advisory committee felt funds were appropriately allocated; does not explain how the sub-question on theory of change will be addressed or answered; research design was flexible and enabled adaptation (expanded from 12 focal organizations to 33 projects, accommodating group interviews, discarding of photo-voice method, appropriate use of a recorder); some informants noted researcher accommodated unexpected invitations to various events or meetings to meet stakeholders/end-use recipients; most informants did not have suggestions for what could have been done differently – only suggested more time in the field or facilitating a forum to bring organizations together, which are outside the bounds of a doctoral project
	Adequate competencies	2	2	2	2	2	Researcher had extensive prior experience in the international development and philanthropic sector; dissertation reflects on methods’ implications for bias and researcher’s ‘outsider’ positionality; translators were hired to support data collection from end-use recipients; research proposal, dissertation, and researcher interview briefly reflected on bias/implications of translators’ involvement (e.g. withholding information, political agenda)
	Research approach fits purpose	1	1	1	1	1	Approach is justified in relation to the context and research objectives (e.g. case study approach, second round of interviews); rationale for different methodologies is explicit, though lacking in terms of discussion on disciplines and epistemologies (researcher interview conveyed appreciation of diverse perspectives gained through the DSocSci cohort) and interrogation of methods and what they brought together was insufficient; discussion on conflicts (but reflecting more on adaptation) is brief
	Appropriate method	2	2	2	1	1.75	Methods are fit to purpose, clearly described, and transparent; methods were systematic yet adaptable to meet the needs of participants or accommodate cultural sensitivity; methods and adaptations, including reflections on the implications of each, are transparent (e.g. abandonment of photo-voice, group interviews); informants reflected positively on the methods and their rigour
	Clearly presented argument	1	1	1	1	1	Results are clearly presented; analyses are comprehensive and logically connected to the conclusions and recommendations; alternative explanations are not explored
Transferability and generalizability of the findings	1	2	2	1	1.5	A statement is included in the dissertation recognizing the possible limit to generalizability of the findings of a single case study; transferability is not discussed in sufficient detail (what, how, and extent of transferability); transferability of findings to other contexts is evident from various outputs of the research; findings have been transferred and applied in other contexts by the PI, participants, and practitioners; informants believed the findings to be widely transferable
	Limitations stated	1	1	1	1	1	The limitations section in the dissertation is underdeveloped (one sentence is given to exploratory nature of the project and how findings are not generalizable, but this could have been explained in more detail); strengths and disadvantages (shortcomings) of methods selected are discussed in the methodology section; the research proposal includes brief reflection on limitations
	Ongoing reflexivity and monitoring	2	2	1	2	1.75	Researcher self-reflection is mentioned as part of the research process, but not discussed in detail in the dissertation (some indications from various e-mail correspondence to the advisory committee and mention of how interviews stimulated additional literature review); changes and adaptations made to methods/approach are transparent and justified (e.g. photo-voice method)
Legitimacy	Disclosure of perspective	2	1	1	1	1.25	Potential bias is briefly mentioned in the dissertation, but the PI is clearly aware of their positionality based on other documentation (admissions proposal, research proposal, book chapter); sources of support (funding, collaboration with DAC) are noted; however, bias and positionality are not discussed in terms of the effect on the project results (only in terms of how funding supported deeper inquiry)
	Effective collaboration	1	1	1	1	1	Interviews conveyed that the internship with DAC for the Mitacs funding and collaboration to funnel/apply findings to the organization was appropriate and clearly agreed upon (however, this was not documented within available project documentation); researcher interview relayed collaborative efforts with participating organizations to provide pro bono services/support; there was potential for more intentional collaboration with participating organizations that would have mutual benefits, but this was not capitalized based on the exploratory nature of the project; collaboration between the PI and the advisory committee was discussed positively
	Genuine and explicit inclusion	1	2	1	1	1.25	Project involved multiple participants from 33 projects, representing over 20 organizations in Moshi – dissertation describes the different characteristics of these organizations (donor country of origin, project focus, size, age); project’s focus on voice of recipients of private development aid clearly distinguishes between end-use recipients and aid implementers; dissertation also distinguishes between the cultural background of local Tanzanian, ex-patriate, and foreign implementers; project also collected data from private donors; second round of interviews functioned to validate accurate representation of the findings; roles of advisory committee, translators, and other support (e.g. transcription, draft editors, editing for SSHRC Storytelling video) are briefly mentioned in the acknowledgements; interview approach had greater emphasis on collecting information than co-generation of knowledge
	Research is ethical	2	2	2	2	2	Project received ethical approval by RRU Research Ethics Board; project sought and received approval from the Tanzanian Commission for Science and Technology to conduct the research; dissertation outlines in detail the steps taken to obtain consent and ensure anonymity of participants; informants reflected on the PI’s commitment to confidentiality and anonymity, which is maintained during this evaluation; dissertation outlines instances of ethical challenges and how they were solved (e.g. use of a recorder, redacted information/turned off the recorder, group interviews, photo-voice method, multinational merger, cultural appropriateness, etc.)
Effectiveness	Research builds social capacity	2	2	1	1	1.5	Some participants noted they learned from the interview discussions and findings (e.g. unintended consequences), while others did not; several informants noted that the findings confirmed/reinforced that they were on the right path; practitioner noted that the research “opened up a dialogue within the NGO world in the organization of accountability”; capacity-building was not explicitly built into the project design, though there is potential that pro bono services supported organizational capacity-building; participant, advisory committee, and practitioner informants believed the doctoral experience and project supported the PI’s knowledge and capacity-building in the sector; some participants/organizations have changed their practices as a result of the research findings
	Contribution to knowledge	2	2	2	2	2	Informants noted that the research uncovered relevant practice gaps they were facing and contributed to the academic literature; some participants noted they learned from the interview discussions and findings (e.g. unintended consequences), while others did not; participants, advisory committee, and practitioners described the findings as “useful” and “lessons to be learned”; researcher disseminated findings to both academic and practitioner target audiences (via conferences, magazine publications, website, SSHRC video, researcher’s consulting, etc.); participants and practitioners shared the findings through their networks
	Practical application	2	2	2	2	2	Intentionality for practical application is discussed by informants; findings have been applied by some participants/organizations (e.g. boards have adjusted the focus of their discussions, added question on unintended consequences into project proposal and monitoring processes); researcher has applied findings in own work and consultancies (e.g. Transform International, Lalafofofo, CAGP, SOVI-BCCIC); findings applied by practitioners (e.g. DAC, Lalafofofo, Transform International)
	Significant outcome	1	2	1	2	1.5	All ten end-of-project outcomes were fully or partially realized with clear project contributions; some organizations have changed their practices as a result of the research (with intent to become more effective); organizational changes are likely to support positive social changes through their work

Table A12.

Individual evaluator and average scoring of SNDP for all QAF criteria, with justifications for the score allocated

Principle	Criteria	E1	E2	E3	E4	Avg.	Justification/Comments
Relevance	Clearly defined socio-ecological context	1	2	1	1	1.25	Dissertation provides a historical, geographical, socio-economic, and socio-cultural description of the problem context; dissertation outlines the complexity of WASH in the Niger Delta and explains compounding factors; identification of academic entry points could have been stronger; dissertation does not explain fully the community, NGO, and policy contexts of the WASH sector in the Niger Delta (would have strengthened the justification for why the research is needed)
	Socially relevant research problem	2	2	2	2	2	Academic and practical gaps are identified and discussed (e.g. negative spillover effects of poor sanitation, appropriateness of WASH technologies for riverine systems are unknown, siloization of Nigeria’s WASH sector); research problem is aligned with Millennium Development Goal 7; informants reflect on relevance and value of study; practical application of the findings are considered and discussed (e.g. PIND)
	Engagement with problem context	2	2	2	1	1.75	PI interacted sufficiently with the problem context to gain a breadth and depth of understanding (via personal experience as a WASH practitioner in Africa, fieldwork engaged a range of system actors); PI was well positioned to influence the context (access to Nigerian WASH networks, working at PIND), but unclear the extent to which this was leveraged (e.g. problem context engagement versus information extraction)
	Explicit theory of change	1	1	1	1	1	Not explicit or documented, but implicit in statements of hoped or intended changes; dissertation indicates opportunities to influence change via PIND or other mechanisms, though this is briefly discussed
	Relevant research objective and design	1	1	1	1	1	Singular objective is weak (lacks specificity) and reads more as a statement of interest; research design is relevant and appropriate to the problem context and exploratory nature of the project; research accounts for and accommodates community needs/values in the design of the approach on sensitive and taboo topic
	Appropriate project implementation	1	2	1	1	1.25	Exploratory approach appropriate considering the scope of the state objective; PI had ongoing engagement with system actors; unclear how new knowledge or reflections made during the research process were incorporated (not documented)
	Effective communication	1	1	1	1	1	No discussion is given on how communications during the research process were planned; research advisory committee commended the PI’s communication of progress and feedback; multiple dissemination strategies were planned; communication of results was perceived to be accessible and well-organized by informants, though there was low awareness of results amongst informants and other actors
Credibility	Broad preparation	1	2	1	1	1.25	Documentation conveys understanding from multiple disciplines, though the integration this knowledge could have been strengthened; various theories and models of behaviour change were reviewed; in-depth understanding of technological options available for the Niger Delta were reviewed and described; the research does not draw on enough previous research from other riverine contexts (only mentioned in passing); engagement with knowledge of former interventions in the Niger Delta was missing
	Clear research problem definition	1	1	1	1	1	The research problem is stated in the dissertation, though broad; the grounding of the research problem in academic literature is vague
	Clear research question	1	1	1	1	1	The research question is stated in the dissertation; appropriateness of the research question was questioned owing to its broadness; some informants critiqued the framing of the research question, feeling that it did not lend well for the project to contribute to a growth of understanding in any one area
	Objectives stated and met	1	1	1	1	1	A singular objective is stated and appears to have been met; however, the objective is not logically or appropriately related to the problem context; a better formulation of objectives would have aided the structure and purpose of the project
	Feasible research project	1	1	1	2	1.25	The design and resources were appropriate to the objective as stated; the project was supported by sufficient funding (Mitacs Fellowship) for fieldwork and hiring of enumerators; PI drew on existing networks through PIND and WASH practitioners; resilience of the design to accommodate opportunities or challenges is not discussed
	Adequate competencies	2	2	2	2	2	The PI had extensive prior field experience and knowledge of Nigeria and the WASH sector; enumerators were given training to conduct the survey; informants commented on the PI’s passion for the work
	Research approach fits purpose	1	1	1	1	1	Exploratory approach fits the stated objective; rationale is given for the approach taken to collect data from the communities (accounting for stigma associated with the topic); there is clear discussion for the inclusion of methodologies, but the selection of disciplines and how they were integrated is not discussed; no discussion of paradoxes or conflicts
	Appropriate method	1	2	1	1	1.25	Mixed methods are clearly described, systematic, and fit to the exploratory nature of the research; however, missed opportunity in the analysis to fully leverage the data collected indicates inappropriateness of the methods
	Clearly presented argument	1	1	1	1	1	Results are clearly presented in both narrative and diagrams; analyses do not demonstrate synthesis of results, instead reading as a list of anecdotes and informant opinions; missed opportunity to draw correlations between the data for a strong argument; the recommendations are not logically connected to the results; alternative explanations are not explored
	Transferability and generalizability of the findings	1	1	1	1	1	Findings are too specific to be transferable; informants thought the research was transferable, as sanitation is a global issue; methods would be transferable, but this is not discussed
	Limitations stated	0	0	0	0	0	Only one brief section of the dissertation acknowledges limited data availability and difficulty in corroborating self-reported qualitative research; limitations are not meaningfully discussed in terms of the implications of results; most important limitations are not addressed
	Ongoing reflexivity and monitoring	1	1	1	0	0.75	Dissertation mentions the importance of reflexivity, but it is not clear the extent to which or how reflection was done; one informant noted the survey underwent significant re-design, but this is not documented in the dissertation
Legitimacy	Disclosure of perspective	0	0	1	0	0.25	Dissertation indicates an awareness of personal biases, but these biases are not openly explained or discussed in terms of the implications for the research; researcher positionality is not discussed; Mitacs funding is acknowledged; partnership with PIND is noted, but connection with Shell and Chevron are not disclosed; notes some participants were known to the PI from own work in the region
	Effective collaboration	1	0	1	0	0.5	The research is not a collaborative piece of work (evidence indicates extractive relationship); there was scope for the research to be more collaborative through the PI’s professional connections to PIND and other WASH practitioners (impression that the PI separated the doctoral work from PIND work); members of the advisory committee reflected positively on their relationships with the PI
	Genuine and explicit inclusion	1	1	1	1	1	A range of system actors were involved in the research, but specifics of their inclusion is not described; steps were taken to engage communities in an appropriate way (being sensitive to their needs and values), but not clear if done for all system actors; indications that the PI made efforts to establish a positive relationship with participants to build trust; creation of the WhatsApp group facilitated a platform for cross-pollination of ideas, perspectives, and experiences amongst WASH practitioners
	Research is ethical	1	2	2	2	1.75	Project received ethical approval by RRU Research Ethics Board; anonymity of participants is preserved; consideration is given on how to ethically engage communities on a sensitive or taboo topic; ethical importance of reflexivity is mentioned
Effectiveness	Research builds social capacity	1	1	1	1	1	Research capacity-building of the PI was high; enumerators received training, but their research capacity is not considered; scope for more intentional capacity-building via the project was a missed opportunity; indications from informants that they think more deeply about the topic and related issues as a result of the research (i.e. how to support development in the Niger Delta); enhanced knowledge-sharing and learning occurring in the Nigerian WASH sector, but unclear if this is a contribution of the project
	Contribution to knowledge	1	1	1	1	1	Contributed to PI’s knowledge; scope for more was a missed opportunity; research not thought to have contributed to new knowledge, but supported bridging connections in existing knowledge and expanding on what is already known
	Practical application	1	1	1	1	1	No innovations were developed from the research; recommendations were intended for uptake, but unclear to what extent this happened; there are some indications of elements taken up by the WASH community, but unclear how and if connected to the project
	Significant outcome	2	2	1	1	1.5	Six out of ten end-of-project outcomes were fully or partially realized, with five indicating clear contribution of the project; potential for more in the future (too early to assess social/economic/environmental benefits), but likely this will result from other processes and interventions in the context

Notes

1. The original criteria, definitions and scoring system have since been revised based on these and other case studies.

2. All terminology should be adjusted & verbally explained so it is appropriate to each interviewee (please record any adaptations in the post-interview notes).

3. It is not necessary to ask all questions to every informant – the list merely illustrates what kind of information we are trying to find out.

4. For the comparative paper, some of the TTP pathways were reframed to allow for cross-project comparisons; notably the Organizational Capacity and Practice pathway was reframed as an Organizational and Sectoral Practice pathway and the Empowerment of Young People pathway was reframed as an Empowerment of Research Participants pathway.

5. For the comparative paper, some of the PDAP pathways were reframed to allow for cross-project comparisons; notably outcomes in the Organizational Capacity and Practice pathway were split to distinguish participant-specific changes in an Empowerment of Research Participants pathway and more general organizational changes in an Organizational and Sectoral Practice pathway.

6. For the comparative paper, some of the SNDP pathways were reframed to allow for cross-project comparisons; notably outcomes in the WASH Sector and Community Development pathway were split to distinguish participant-specific changes in an Empowerment of Research Participants pathway and more general sector-level and PIND-specific changes in an Organizational and Sectoral Practice pathway.