Evaluating Processes, Outputs, and Outcomes to Learn and Improve

aUniversity of California, USA
bRoskilde University, Denmark

Co-Creation for Sustainability

ISBN: 978-1-80043-801-9, eISBN: 978-1-80043-798-2

Publication date: 2 August 2022

Abstract

This chapter insists that evaluation of the process and results of cocreation is a precondition for continuous improvement and helps maintain support from external sponsors and funders. The main benefits of systematic evaluation of cocreation are learning and legitimacy rather than control and allocation. The chapter scrutinizes the two most common evaluation tools, formative and summative evaluation, and finds that they both fail to appreciate the emergent character of cocreation processes. The solution to this problem is to supplement formative and summative evaluation with developmental evaluation, which prompts the participating actors to engage in a critical interrogation of what they are doing, the reasons for doing it, and the results they achieve. Finally, the chapter explains how the commitment of developmental evaluation to using real-time data in the evaluation of change theories can be pursued through a collective impact strategy.

Keywords

Citation

Ansell, C., Sørensen, E. and Torfing, J. (2022), "Evaluating Processes, Outputs, and Outcomes to Learn and Improve", Co-Creation for Sustainability, Emerald Publishing Limited, Leeds, pp. 151-164. https://doi.org/10.1108/978-1-80043-798-220220012

Publisher

:

Emerald Publishing Limited

Copyright © 2022 Christopher Ansell, Eva Sørensen and Jacob Torfing. Published by Emerald Publishing Limited. This work is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this work (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode.

License

This work is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this work (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode.


Why Should We Evaluate Cocreation?

Classical forms of bureaucratic government have always emphasized the need for public managers to ensure that public employees follow legal and administrative rules, operate within budget limits, and deliver services and solutions according to plan. Hence, despite their motivation to use their skills to solve public problems and create public value, public employees must be controlled to make sure that they perform as expected and in line with professional standards. The wave of New Public Management reforms (Hood, 1991) from the 1980s onwards criticized bureaucratic government for merely focusing on frontline workers compliance with bureaucratic rules and regulations and for failing to pay adequate attention to the results that are produced. Following this criticism, New Public Management recommended that bureaucratic rules and regulation be relaxed and more freedom given to local agencies and employees so that they could work more flexibly to improve efficiency, deliver effective solutions, and improve results (Osborne & Gaebler, 1992). The relaxation of compliance control was to be compensated by a more intense evaluation of the performance of local agencies and employees based on rigorous assessment of outputs and outcomes, regular reporting to central auditing agencies, and the use of incentives to affect future performance (Barber, 2007). Since New Public Management also recommended that public services were outsourced to private firms, performance evaluation was not only directed toward public agencies and their employees but also targeted private service contractors.

While evaluation has always been an integral part of the policy process, New Public Management has greatly enhanced the public sector's focus on evaluation. The number of public, semipublic, and independent auditing and evaluation agencies has mushroomed, and evaluation has become a regular and increasingly professionalized activity. New Public Management is mostly concerned with evaluation because it helps to control frontline personnel and prevents them from shirking and provides valuable information that can be used when making decisions about the allocation of public funds, for example, by shutting down low-performing agencies and boosting high performers. However, evaluation may also serve other noble goals such as enhancing learning and legitimacy. Evaluation of processes, outputs, and outcomes may spur learning by providing feedback that stimulates critical reflection, especially if evaluation is used in the early phases of a project and there is time to improve the process and correct errors, or if a project involves the design and testing of prototypes that are revised before they are upscaled. Evaluation may also enhance legitimacy by enabling project managers to produce a carefully documented account of what has been done and what has been achieved, thus ensuring public transparency and convincing sponsors, public authorities, and local communities that the money and resources have been spent well.

Learning and legitimacy are the primary benefits when cocreation projects are evaluated. Nevertheless, some people might object to the idea of subjecting cocreation to evaluation, either because they are afraid that the deployment of systematic and rigorous evaluation procedures may hamper the creativity of social entrepreneurs, or because evaluation appears to be a waste of time and energy since all the participants in cocreation are good hearted people who are doing their utmost to save the world. While we agree that a hard-handed, rigorous evaluation performed by external auditors may scare off the private, for-profit, and nonprofit actors who are participating voluntarily in cocreation projects, we shall insist that evaluation is strictly necessary for spurring learning and enhancing legitimacy. Social entrepreneurs and other actors involved in cocreation need to know whether the collaborative process is organized in ways that stimulate innovation and build joint ownership and whether their more or less innovative solutions solve the problem at hand and preferably without generating unforeseen negative side effects.

Hence, if an evaluation of cocreation shows that the process is conducive for mobilizing ideas and resources and fostering innovative solutions that work in practice, the actors involved in the cocreation process can pat themselves on the back, recharge their batteries, and raise their ambitions. Conversely, if an evaluation detects problems and unexploited opportunities or finds that proposed solutions are not hitting the target, the involved actors may ponder how to improve the process through piecemeal adjustments and changes and how to redefine the problem and revise the action theory that projects and explains the likely impact of a particular solution. In both situations, the cocreating actors come out of the evaluation process as winners. In sum, the question is not so much whether or not to evaluate cocreation, but rather how to do it and who should do it. As such, the impact of purpose-built cocreation processes may be enhanced as a result of evaluation conducted by the involved actors. The dynamic relation between process, impact, and evaluation is illustrated in Fig. 11.1.

Fig. 11.1. 
The Dynamic Relationship between Process, Impact, and Evaluation.

Fig. 11.1.

The Dynamic Relationship between Process, Impact, and Evaluation.

This chapter takes a critical look at different forms of evaluation arguing that cocreation may benefit from a combination of process evaluation, developmental evaluation, and the use of collective impact studies. More traditional evaluation tools such as formative and summative evaluation may also be applied, but as we shall see, these tools have problems dealing with the emergent character of co-creation.

Process Evaluation

Evaluation is a key to sparking ongoing learning about and improvement of cocreation processes in order to enhance their quality and impact while providing a solid account to external actors of how money is spent. Process evaluation is an ongoing activity that allows participants to assess and perhaps influence the factors that either promote or inhibit collaboration and the search for innovative solutions that can improve the conditions for social, economic, and environmental sustainability.

An easy way of evaluating collaborative processes that aim to develop innovative solutions is to use the self-evaluation tool proposed by Borden and Perkins (1999). This tool lists no less than 12 factors that may spur or hamper collaboration. Our adapted version of the collaboration checklist list is provided in Table 11.1.

Table 11.1.

The Collaboration Checklist.

  1. Sustainable participation: The collaboration has a plan for sustaining membership and resources. This involves membership guidelines about how to become a member, the expectations for the involvement and contribution of members, how members are replaced should they want to leave, and how new members are recruited if necessary.

  2. Communication: The collaboration has open and clear communication. There is an established process for communication in and between meetings.

  3. Research and data collection: The collaboration has conducted a proper assessment of local needs and has obtained information to establish its goals.

  4. Political climate: The history and political environment surrounding discussions and decision-making in the collaboration is positive and supports cocreation of new and innovative solutions.

  5. Policies, laws, and regulations: The collaboration has managed to change policies, laws, and/or regulations to allow the collaboration to function effectively.

  6. Resources: The collaboration has access to needed resources. Resources refer to four types of capital: Environmental, in-kind, financial, and human.

  7. Catalysts: There is a clear idea about the problems that call for collaboration and the mutual dependence between the actors that prompts them to engage in resource exchange.

  8. Track record for collaboration: The community has a history of working cooperatively to solve pressing problems, and there are positive experiences to draw upon.

  9. Connectedness: Members of this collaboration are well-connected and have established informal and formal networks at all levels that allow experiences, ideas, and resources to flow freely.

  10. Leadership: One or more leaders facilitate and support team building and capitalize upon diversity and individual, group, and organizational strengths.

  11. Community development: The local community has been mobilized to address important issues. There is a communication system and formal information channels that permit the joint exploration of issues, goals, and objectives.

  12. Understanding community: The collaboration understands the community, including its people, cultures, values, and habits.

Process evaluation based on the collaboration checklist is very simple. The members of the collaboration are asked to assess the 12 factors on a five-point Likert scale ranging from strongly agree (1) to strongly disagree (5). If the total average score of collaboration is between 0 and 30 points, most of the conditions for successful cocreation are in place and there is no reason to worry. If the total average score is higher, there is room for improvement and the actors participating in the cocreation process should discuss how to improve the conditions for successful collaboration. Some of the above factors are easier to influence than others. However, the important thing is not to eliminate all barriers to collaboration, but rather to constantly do what is possible to improve the conditions for successful cocreation.

Collaborative platforms supporting networked cocreation processes may be evaluated in terms of their capacity for knowledge aggregation, creativity, and decision-making. Mačiulienė and Skaržauskienė (2016) have studied 30 digitally supported collaboration platforms in Lithuania and find that six factors are worth evaluating in order to assess the effectiveness of collaborative platforms. The six factors are summarized in Table 11.2.

Table 11.2.

Evaluating Collaborative Platforms Supporting Cocreation Processes.

  1. The degree of openness and flexibility reflects the diversity of the participants and is important for learning about needs and experiences that can stimulate creative problem-solving.

  2. The degree of diversity concerns the adaptability of the platform to different groups and the different opportunities for disseminating and sharing knowledge, both of which support mutual learning.

  3. The degree of interdependence relates to the opportunities for exchanging ideas about how problems can be solved and commenting on the content of these ideas.

  4. The degree of decentralization reflects the existence of different forms of decision-making and the presence of equal rights for all participants to express their views and participate in decision-making. Involvement in decision-making is important to build common ownership for solutions.

  5. The degree of transparency refers to the existence of a transparent organizational structure, clear rules and norms for self-organization, and a distributed memory.

  6. The degree of security reflects the ability of the participants to anonymously voice ideas and opinions and the level of personal data protection. Security is important in order to facilitate dissent and protect the privacy of the participants.

The evaluation of how cocreation processes can be effectively supported by collaborative platforms is still in its infancy, and there are no clear indications of the causal effects of the different factors cited above. Nevertheless, people who are in charge of designing collaborative platforms that aim to support cocreation processes may try to enhance the presence of the six factors mentioned above as a purposive way of stimulating learning, innovation and decision-making and evaluating outcomes. Platform design will often involve a good deal of experimentation until there is a good fit between the platform and the cocreation processes it is meant to support and enhance.

Formative Evaluation

When relevant and affected actors from the public and private sector are brought together to engage in a cocreation process, they will plan a broad range of activities that will help them to find a suitable solution to the problem at hand and thus achieve one or more Sustainable Development Goals (SDGs). The planned activities may include fact finding missions, problem definition workshops, public hearings with discussion of tentative solutions, consultation of external actors with special forms of expertise, small-scale testing of prototypes, meetings with organizations that may finance the upscaled solution, drafting of a comprehensive implementation plan, etc. The portfolio of planned activities will vary from project to project. While a few activities may be canceled and some new activities may be added, the participants and their sponsors will want to keep track of whether the bulk of their planned activities are realized and whether they contribute to driving the cocreation process forward. The participants may even define deadlines and milestones that must be reached in order to keep the momentum of the group intact and make progress toward the development of a solution that hits the target and enjoys widespread support.

Cocreating actors who are seeking to keep track of what they have accomplished and how their efforts bring them closer to goal achievement may consider using formative evaluation (Brown & Gerhardt, 2002). Formative evaluation aims to take stock of a particular project or process through regular assessments of whether planned activities have been carried out and milestones are reached. In addition, it aims to solicit feedback on past activities from users and relevant stakeholders in order to improve the quality and impact of future activities. Finally, it seeks to identify the strengths and weaknesses of the project and to delineate the room for improving the project in order to increase its chances of realizing the overall goals at the end of the process (Beyer, 1995). As such, formative evaluation aims to prepare projects or processes for the final assessment of their achievements by measuring their performance against the expectations of the participants and the plans they have made for what to accomplish, how, and when.

The procedure to follow when conducting a formative evaluation of a collaborative process is relatively simple. First, draw up a complete list of planned activities and identify the related deadlines and milestones. Second, check whether the activities have been carried out as planned and met their deadlines, and if not, find out why this is so. Third, solicit feedback on the completed activities from internal and external actors, for instance, through focus group interviews or mini-surveys. Fourth, summarize the feedback and facilitate joint discussion of the lessons to be drawn. Finally, use the feedback and lessons to re-design future activities in order to improve their quality and impact. This procedure might be repeated at regular intervals to effectively revise the cocreation process and its outputs in ways that resonate with the expectations of the participants and increase the chance of fostering a desirable outcome.

Lessons about how to work collaboratively to achieve one or more SDGs may be shared with other interested actors. To that end, in 2016, the Finnish government asked the members of the National Commission on Sustainable Development and the Development Policy Committee how they intended to implement Agenda 2030 in the work they carry out within their own organization and in collaboration with other organizations. They were also asked what kind of new or innovative activities they had developed. This simple survey generated a large number of responses that were subsequently shared in order to inspire other actors to adopt new ways of working.

On a final note, it is important to understand that formative evaluation is not about reaching the summit, but rather about improving the process of getting there. Experienced hikers tend to stop at regular intervals in order to check where they are, evaluate the path they have taken, and take stock of their physical condition and energy level. Based on this brief evaluation, they make decisions about how to approach the new terrain. They may decide to slow their pace and choose a less strenuous route to maintain energy and to drink more water, change their clothing, and adjust their backpack to improve their performance. Hence, formative evaluation aims to improve the journey to ensure steady progress. Otherwise, there is no chance of reaching the summit.

Summative Evaluation

Hikers may not be satisfied with having had a pleasant journey if they do not reach their final destination and achieve the goal they set out to attain. The same goes for collaborative governance processes. It is nice enough for the involved actors to be able to look back at a series of well-executed and worthwhile activities, but if the pressing problems that brought the actors together remain unsolved and the common goals are not achieved, all the hard efforts are not worth much. In problem-driven cocreation processes, goal attainment is crucial and in order to assess the extent to which the jointly formulated goals have been reached, the participating actors may consider to use summative evaluation.

Whereas formative evaluation is an ongoing activity that focuses on whether and how planned outputs are delivered, summative evaluation takes place at the end of the project or process and aims to assess the degree to which a given set of predefined goals have been achieved. A project may have several important goals that are fulfilled to a greater or lesser extent, sometimes displaying trade-offs whereby the realization of one goal negatively influences the achievement of another goal. A goal hierarchy may exist and might help to produce an overall assessment of whether the project as a whole has been successful.

Summative evaluation is not only comparing a pre-given set of goals with available data in order to measure the degree of goal attainment. It also aims to explore whether the final outcomes can be ascribed to the outputs of the project, and if so, what the causal mechanism relating outputs to certain outcomes is. Finally, summative evaluation also seeks to determine the conditions under which project activities and interventions have led to the realization of one or more goals in order to probe the possibility for generalization and scaling. The ultimate purpose of summative evaluation is to hold the actors involved in the process to account for their achievements and learn from the solution they produced.

The main barrier to summative evaluation in relation to the SDGs is the lack of precise operationalization of the goals and the lack of accurate data permitting measurement of progress. The UN has helpfully established 161 indicators designed to measure SDG progress. However, if national governments and their statistical agencies do not collect data relevant to these indicators, it becomes difficult to evaluate progress. Therefore, there is a need for each country to tailor UN indicators to national contexts. In Denmark, the 2030 Panel established by the national parliament has collaborated with Danish Statistics, Deloitte Consulting, university professors, and scores of private companies and organizations to find quantitative measures of the SDGs that seem relevant in the Danish context. This work has resulted in the report Our Goals (Danish Statistics, 2020), which contains 197 measures that allow assessment of whether Denmark as a country is making progress toward the achievement of the SDGs. The challenge is that even with goals tailored to the national context, it may be difficult to evaluate the contribution of local projects to these macroscopic indicators. Nevertheless, national indicators can provide important targets for local projects.

Other countries have established statistic platforms supporting the measurement of goal achievement in relation to the SDGs. One example is New Zealand, which has developed a new statistical policy indicator framework that goes beyond traditional economic indicators to assess the state of well-being of all groups in the population. This framework is to replace GDP as the lodestar for national policy making and will provide ongoing feedback to decision-makers at all levels about the effects of policies on the well-being of the population and its natural environment. Another example is Armenia, which has established an SDG statistical platform to build data gathering capacity and routines in support of decision-making and goal assessment. Given the low level of statistical capacity only a few years ago, this platform marks a significant addition to the Armenian public sector's steering capacity. It also helps to build trust with the international community and foreign private investors.

Formative and summative evaluation is often used in tandem with formative evaluation helping to prepare a project for summative evaluation. The combination of the two evaluation methods is particularly useful for “blueprint” projects that have clear, predefined goals, involve implementation of standardized preplanned program activities, and have a definite ending point. None of these requirements are fulfilled in cocreation processes that stress curiosity, creativity, and deliberation. The emergent character of cocreation means that both goals and activities are subject to constant reformulation. Moreover, the wicked character of many sustainability problems means that the attempt to solve them is an ongoing activity with no clear end date. The actors are engaged in a continuous process of innovation and improvement, and projects tend to extend beyond what was originally planned, perhaps in new and different forms, or as parts of a larger venture. The lack of predefined goals and activities and a definite ending point limits the usage of formative and summative evaluation and calls for an alternative evaluation method that better fits the emergent cocreation processes.

Developmental Evaluation

Developmental evaluation offers an alternative to formative and summative evaluation that better aligns with the emergent character of cocreation. As such, it provides a mechanism for stimulating learning and adaptation in cocreation processes that aim to advance one or more SDGs (Feinstein, 2019; Reynolds, Gates, Hummelbrunner, Marra, & Williams, 2016). As indicated in Fig. 11.2, formative and summative evaluation are linear evaluation techniques taking place either at regular intervals or at the end of the process, whereas developmental evaluation is an ongoing activity that tends to force the actors involved in cocreation to move back and forth between goals, activities, outputs, and outcomes in order foster mutual adjustments as a result of mutual learning.

Fig. 11.2. 
Formative and Summative Evaluation Compared with Developmental Evaluation.

Fig. 11.2.

Formative and Summative Evaluation Compared with Developmental Evaluation.

According to Patton, “developmental evaluation is designed to be congruent with and to nurture developmental, emergent, innovative and transformative processes” (2010, p. 7). Developmental evaluation recommends that cocreators undertake a continuous reality-testing of their changing assumptions, propositions, and ideas and thus offers a strategy for evaluation of cocreation that is compatible with its emergent character, which derives from the fact that problems, goals, activities, and solutions are shaped and reshaped through processes of mutual learning, collaborative innovation, and the chance discovery of new activities and solutions (Patton, 2010).

Developmental evaluation takes place throughout the steadily evolving cocreation process and is usually carried out by one or more team members who encourage their fellow collaborators to ask critical evaluative questions. Are we sure that we understand what the problem is? What is it exactly that we want to accomplish? Why are we doing things in this way? Do our assumptions about the preferred solution hold up? How do we know that? Is the proposed solution sufficiently robust? Will we be able to muster broad support for its realization? Have we secured adequate funding and financing? Do we have an efficient procedure for collaborative adaptation of the new solutions on the ground? Answering these and other evaluative questions based on a systematic collection and analysis of data will help to spur learning and innovation and thus enhance the problem-solving impact of cocreation.

In developmental evaluation, the cocreating actors test whether their causal assumptions about the sources of a problem and its negative effects hold in the face of a closer scrutiny that may involve empirical studies, consultation of experts, and empathetic involvement with those affected by the problem. They identify and seek to remove barriers to mutual learning, critically examine the range of possible options and the prospect for combining these into something new and promising, and they explore whether joint solutions produce the expected effects. Finally, they interpret possible signs that the problem is diminished due to the implementation of new solutions and critically review how new and unforeseen challenges are dealt with.

Developmental evaluation critically interrogates the goals, ideas, and propositions that emerge in the process of creative problem-solving by thinking through their implications and testing them through real-life experiments and systematic data collection. Goals that have no relevance for the attempt to solve the problem, ideas that cannot be realized, and solutions that do not produce the expected results are challenged, reformulated, and submitted to a new test. As such, developmental evaluation confirms the idea that the development of robust solutions through active engagement of relevant and affected actors requires iterative rounds of goal formation, trying out solutions, and evaluation of impacts (see Andrews, Pritchett, & Woolcock, 2013).

Developmental evaluation does not assume that there is an ultimate solution or outcome of cocreation processes. Rather, it is asserted that outcomes are always provisional, conditioned by turbulent environments and subject to ongoing innovation. This assertion places learning at the heart of developmental evaluation. Its purpose is not to hold responsible actors to account for their action and inactions, but rather to learn more about the collaborative process and the attempt to define problems, design solutions, and ensure their practical realization (Mockbee & Newsham, 2013). Developmental evaluation is a learning-stimulating evaluation tool that invites participation actors to critically scrutinize what is working and what is not working in order to reformulate the goals and improve activities, outputs, and outcomes (Patton, 2010).

To illustrate the argument about the benefits accruing from utilizing developmental evaluation, let us briefly summarize the experiences from the Minnesota-based McKnight Foundation's Collaborative Crop Research Program (CCRP) (Moore & Cady, 2016). To fight the global food crisis (SDG 2), the CCRP promotes interdisciplinary research on plant science aimed at producing greater crop yields. Founded by private donors, it brings together over a 100 agricultural research projects from Africa and South America in an inclusive, multi-actor decision-making process involving scores of researchers, policymakers, farmers, and civil society organizations. The evaluators and program leaders work closely together to develop an adaptive action framework for evaluating activities, results, and impacts. Extensive work has been done to create a culture where people feel safe to seize every opportunity to ask: “What? So what? Now what?” The new practice of asking evaluative questions is supported by the development of a new data system, technical and conceptual skills, and procedures for communicating the interpretations of evaluative data. As a result of these transformations, the participants have gradually come to perceive the program and its projects as being in a “state of becoming” where new insights, activities, and results are regarded as work in progress. The overall experience with developmental evaluation is positive, as the embrace of emergence has spurred collaborative innovation while retaining fidelity to the overall mission.

Developmental evaluation of cocreation is sometimes met by the objection that it is difficult to get busy, action-oriented, and impact-driven actors to spend time reflecting on the process and impact of cocreation and the need to transform the modus operandi, modify the preferred solution and seek to make systemic changes. Our response to this recurrent objection is that critical reflection and mutual learning are not external to but key ingredients of cocreation that rely on carefully orchestrated learning processes to foster innovative public value outcomes.

When a network of cocreating actors attempts to assess its collaborative processes and joint activities, it may use the process evaluation checklist cited above. The evaluation of the outcomes of cocreation appears to be more complicated. First, despite joint agreement on the overall goal, the actors may have different views of what constitutes a benefit or a cost vis-à-vis the common goal. They may have different normative belief systems that influence their judgments and their evaluation of the results and impacts of cocreation may reflect their relative net gains.

Second, cocreation projects often have intangible goals and produce intangible outcomes such as public safety, resilient communities, human wellbeing, holistic health care, sustainable agriculture, democratic empowerment, democratic legitimacy, etc. that are much more difficult to measure than the quantity and quality of public services or the growth in GDP. The measurement problem is intensified if the outcomes are only detectable in the longer term (Loeffler & Bovaird, 2018, p. 272).

Third, the tendency of cocreation projects to focus on broader societal benefits may hide the public costs that public leaders and employees may incur. These costs may include: the increasing time spent setting up the framework for cocreation and participating in and managing processes; large investments in ICT-enabled platforms and tools supporting cocreation; heightened costs of informing, instructing and training private actors engaged in cocreation; and rising expenditure resulting from growing ambitions and more intensive forms of cocreation and co-production (Loeffler & Bovaird, 2018, pp. 276–277).

A final complication when measuring the results of cocreation processes is the difficulty that emerges when assessing the trade-offs between different goals or between benefits and costs. Although there might be some form of joint leadership, cocreation arenas do not contain a hierarchical authority than can legitimately settle disputes and adjudicate the priorities that are made.

Despite these complications, developmental evaluation offers a welcome alternative to formative and summative evaluation that is compatible with emergent forms of cocreation. Systematic application of developmental evaluation is important as it helps to ensure the legitimacy of cocreation in the context of political demands that public money and managerial resources are not wasted on time-consuming collaborative processes that are nice and cozy but fail to generate results. It may not provide an ultimate verdict about whether and why a particular cocreation project was successful or unsuccessful, but it offers a fine-grained analysis of what proves to be working well and how it can be further improved.

Fast Learning From Collective Impact

Despite the strong commitment of developmental evaluation to testing the actual impact of the current theory of change against purposefully collected data, there is a risk that the attempt to facilitate fast learning may drown in the myriad of evaluative questions that are asked and answered in relation to goals, activities, outputs, and outcomes. Failing to conduct ongoing impact studies that rigorously document the effect of cocreated solutions is a huge problem. Not only will the cocreating actors be in the dark as to whether their efforts are leading them in the right direction, but they will also be unable to report progress and learning to their sponsors and funders who are likely to require regular impact reports in return for continued political and economic support. To avoid these problems, this final section looks at how local cocreation projects can benefit from insights derived from collective impact studies.

Like developmental evaluation, collective impact embraces emergence by focusing on the ongoing progression, discovery, and learning that seem to accelerate social change without necessarily requiring breakthrough innovations and vastly increased funding (Kania & Kramer, 2013). Collective impact aims to spur fast learning obtained through ongoing feedback loops that use carefully collected data to detect changes and spur joint discussions of what is happening, how it is happening, and why it is happening. This allows the involved actors to draw lessons about how to make use of changing conditions to solve pressing problems and convert these lessons into collective action informed by slightly different ideas and visions. In practice, collective impact comes down to a dedicated effort to communicate real-time data through dashboards, weekly outcome diaries, or the production of running narrative that documents how the work is unfolding (Kania & Kramer, 2013).

The strength of the collective impact framework is that it clearly specifies the conditions for using continuous feedback and fast learning as drivers of creative problem-solving (Kania & Kramer, 2011). First, all participants must have a common agenda, including a common understanding of the problems and a shared approach to solving them through collective action. Without a common vision consisting of one or more goals and some ideas about how they can be accomplished, there can be no alignment and the collaborative efforts will lack purpose and have no impact. Moreover, if there is a vision, but no signposts, there can be no learning about progress, obstacles, new opportunities, and the need to adapt the modus operandi.

Second, the common agenda must be translated into shared measurement, and impact data must be collected consistently across all participants who should be invited to discuss what they see and what can be learned. The shared measurement has three crucial effects: it helps to align the actors by providing a common object for analysis that encourages joint deliberation; it strengthens horizontal accountability by allowing the actors to hold each other to account for results; and it facilitates mutual learning about current practices, what works, and what needs to be changed.

Third, the results of the fast learning generated through reflection on the ongoing collection and analysis of impact data must be disseminated through continuous communication to all participants so that they can all act in unison on the new insights, whether these point to doing more of the same, or to doing things differently in order to improve the impact. Capturing data-driven learning is not worth much if the lessons learned are not communicated, preferably in a way that stimulates collective action.

Fourth, mutually reinforcing activities must be spurred through efforts to coordinate the distributed actors. Hence, a mutually reinforcing plan of action must be drawn up to ensure that the participating actors respond to new knowledge and adopt new solutions at the same time, thus creating “cascading levels of linked collaboration between cocreation arenas, partner organizations and community members” (Hanleybrown, Kania, & Kramer, 2012).

Finally, all of the above must be supported by a backbone organization that is equipped with staff and specific skills to collect and analyze data, communicate learning points, and coordinate action. The backbone organization may either be a lead organization from the public or private sector or draw its staff from a representative sample of stakeholders. Table 11.3 summarizes the five conditions for fast learning based on ongoing feedback loops.

Table 11.3.

Conditions for Learning in Collective Impact.

Common agenda A shared vision helps to guide learning
Shared measurement Consistent measuring of impact creates joint focus on progress
Continuous communication Communication of learning points facilitates joint action
Mutually reinforcing activities Coordination of distributed action optimizes impact
Backbone support Dedicated staff and special skills drive learning-based evaluation

Source: Adapted from Kania and Kramer (2013).

Like process evaluation, collective impact is an evaluation tool that can be used along with developmental evaluation. These three evaluation approaches focus on real-time rather than retrospective evaluation and aim to spur learning in order to improve the process and outcome of cocreation. In a nutshell, it all boils down to asking two questions: “How can we improve communication across partners?” and “What measures will show that we are making progress” (Weaver, 2014). Answering these fundamental questions will most likely help advance community-based change.