Search results

A meta-evaluation is an assessment of evaluation practices. Meta-evaluations include assessments of validity and usefulness of two or more studies that focus on the same issues. Every performance audit is grounded explicitly or implicitly in one or more theories of program evaluation. A deep understanding of alternative theories of program evaluation is helpful to gain clarity about sound auditing practices. We present a review of several theories of program evaluation.

This study includes a meta-evaluation of seven government audits on the efficiency and effectiveness of tourism departments and programs. The seven tourism-marketing performance audits are program evaluations for: Missouri, North Carolina, Tennessee, Minnesota, Australia, and two for Hawaii. The majority of these audits are negative performance assessments. Similarly, although these audits are more useful than none at all, the central conclusion of the meta-evaluation is that most of these audit reports are inadequate assessments. These audits are too limited in the issues examined; not sufficiently grounded in relevant evaluation theory and practice; and fail to include recommendations, that if implemented, would result in substantial increases in performance.

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.

A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.

Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.

The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.

The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.

As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.

Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.

The purpose of this study is to characterize output and impact evaluation initiatives from the perspective of entrepreneurs and investors of social enterprises (SE) by highlighting their relevance, implementation processes and consequences.

This study adopted a qualitative approach through a multiple case study. This study also analyzed ten SE in five areas and three investors, using semi-structured interviews, codification and content analysis.

Evaluation is still at an initial stage in these businesses, and there is a gap between what the field-specific literature proposes and what firms do. Although entrepreneurs feel that evaluation is essential to demonstrate their value and confirm their purpose and impact, it is not a priority.

Due to the small number of Brazilian enterprises and investors, it is impossible to generalize the results.

The results are relevant to rethink strategies, processes and methodological tools that seek to portray, more realistically and reliably, the evaluation dimensions in SE.

This study highlights the risk regarding the legitimacy of social impact ventures since, without the due causal inferences, it is not easy to prove the impact they claim to achieve. This situation conducts to the weakening of the ecosystem leading to the emptying of the term impact.

The main research contributions address the methodologies, practices and evaluation routines of some Brazilian SE and their investors, presenting the characteristics, perceptions and limitations of the studied ventures.

The purpose of this paper is to describe the evolving IFLA approach to impact evaluation through three of its international programmes: Freedom of Access to Information, Building Strong Library Associations (BSLA) and the International Advocacy Programme (IAP). This review positions these three programmes within the wider discourse of the international evaluation community.

Each of the three programmes is considered in turn to show what they were trying to achieve and how thinking about impact evaluation at IFLA is evolving.

This paper reports key evaluation findings for relevant phases of the BSLA and IAP programmes in general terms.

The views presented are those of the evaluation consultants who advised each of these programmes (and in the cases of BSLA and the IAP conducted the programme evaluations).

The processes described and the conclusions drawn should be of interest to anyone involved in international or national library evaluation, especially of public libraries, library associations and national libraries.

The paper suggests that more systematic impact evaluation of public libraries, library associations and national libraries is necessary to ensure their future survival.

The authors were uniquely placed to see and participate in IFLA impact evaluation discussions over the past decade.

This study aims to develop an ex-post impact evaluation method for public-private partnership projects (PPP) that addresses one of its main difficulties: the multiplicity of perspectives and stakeholders that exist in these projects.

This study first developed a conceptual evaluation framework called ‘Project Success Evaluation Pyramid Model’ (PSEPM) that was designed using a Design Science approach. Second, an impact evaluation method was designed based on the PSEPM. This evaluation method was tested and refined using three PPP projects. This article presents the refined conceptual model PSEPM, the evaluation method and one of the three tests: the PPP toll road.

The impact evaluation method describes how to evaluate the impact of an existing mega project from the perspective of multiple stakeholders and isolate the impact of the procurement method. Its application in practice is demonstrated with a PPP toll road project in Chile, in which 11 stakeholders were identified. Issues such as city segregation, deficient business model, the relevance of infrastructure availability, and deficient service-fee strategy were identified in the project. The evaluation method provides a judgement about the use of the PPP procurement method and about the overall impact of the project, reconciling the stakeholder perspectives.

The use of this approach could give greater clarity and legitimacy to the discussion over the use of PPPs by evaluating existing projects beyond the use of traditional time and cost measurements. Its originality is the systematic recognition and identification of stakeholder motives, purposes, and judgements over a project.

The purpose of this paper is to explore, in literature and practice, the use of ex post impact evaluations within the public–private partnership (PPP) context and understand the major considerations for developing a PPP ex post impact evaluation method.

This paper applies exploratory research through expert interviews from Chile and Australia to describe: the relevance of this kind of evaluation, and the challenges of designing and conducting a PPP ex post evaluation.

The study confirms the lack of a formal method for evaluating the impact of a PPP project. Experts agree on the relevance of performing ex post evaluations to PPP projects and that in practice there is no formal procedure to follow. Among other challenges, experts discussed four general ex post evaluation aims: transparency and accountability, PPP legitimization, industry learning and government agency learning.

This study confirms the gap in knowledge and contributes to the developments of approaches to perform ex post impact evaluation of PPP projects. It also provides several suggestions that need to be addressed when attempting to evaluate PPPs beyond the financial and contractual parameters.

The topic is not fully addressed in the literature, and this study contributes to the initial discussion and development of this evaluation method, which is considered significant for the development of public infrastructure.

Improvement in the evaluation methodologies used in the public policy and development fields has increased the amount of evidence-based information available to decision makers. This helps firms evaluate the impacts of their social investments. However, it is not clear whether the business sector is interested in using these methods. This paper aims to describe the level of interest in, knowledge of and preferences relating to the impact evaluation of corporate social responsibility (CSR) programs by managers in Latin American companies and foundations.

A survey of 115 companies and foundations in 15 countries in Latin America was conducted in 2019.

The results indicated that most respondents believed that quantitative impact evaluation could address concerns about CSR program outcomes. However, monitoring and evaluation were primarily seen to be for tracking program objectives rather than for making strategic decisions about innovations to enhance the achievement of outcomes. Decision-making tended to respond to community demands. The main challenges to increasing the use of impact evaluation were the lack of skills and knowledge of management staff and the methodological complexity of evaluation designs. We conclude that there needs to be increased awareness about: the appropriate understanding of social outcomes; the benefits of evaluation; when impact evaluation is useful; how to prepare an evaluation budget; and the effective use of rigorous evidence to inform program design.

Acceptance by the business sector of quantitative measurement of the social impact of CSR programs will lead to improved outcomes from social investment programs.

The purpose of this present study is to find a scientific method for the evaluation of environmental impacts according to the requirement 4.3.1.

To realize the objectives, the authors worked with a representative sample from certified ISO 14001 organizations. The data aim to identify and evaluate (according to the organization's methodology) significant environmental impacts. In this study, the authors created two models for the evaluation of environmental impacts based on an artificial neural network applied in the pilot organization and compared the results obtained from these models with those obtained by applying an analytic hierarchy process (AHP) method. AHP is part of an multi‐criteria decision making method and provides good multi‐criteria support for decision making for problems that can be structured hierarchically.

This paper presents a new approach that uses a backpropagation neural network to evaluate environmental impacts regardless of the organization type.

This paper presents a unique approach for the reliable and objective evaluation of environmental impacts.

Though we have recently witnessed the “exponential production of digital data to measure, analyze, and predict educational performance” (Salajan & Jules, this volume), there has not been sufficient attention given to the quantitative methods that are used to process and transform this data in order to arrive at findings related to “what works”. This chapter addresses this gap by discussing a range of constraints that affect the main methods used for this purpose, with these methods being known as “impact evaluation.” Specifically, this chapter addresses its purpose, first, by making explicit the methodological assumptions, technical weaknesses, and practical shortcomings of the two main forms of impact evaluation—regression analysis and randomized controlled trials. Although the idea of Big Data and the ability to process it is receiving more attention, the underlying point here is that these new initiatives and advances in data collection are still dependent on methods that have serious limitations. To that end, not only do proponents of Big Data avoid or downplay discussion of the methodological pitfalls of impact evaluation, they also fail to acknowledge the political and organizational dynamics that affect the collection of data. To the extent that such methods will increasingly be used to guide public policy around the globe, it is essential that stakeholders inside and outside education systems are informed about their weaknesses—methodologically and in terms of their inability to take the politics out of policymaking. While the promises of Big Data are seductive, they have not replaced the human element of decision making.

The purpose of this paper is to present a critically reflective account of the process of conducting an impact evaluation of a dance-theatre company’s staged productions and workshops.

There are two main approaches: the introspective critical reflection on the process of performing an impact evaluation; and the drawing/colouring methods used to perform it.

It is more difficult to provide impact evaluations of the soft rather than hard outcomes of publicly funded performance arts. The engagement of third-party evaluators may help overcome the challenge that many “accepted” approaches are outwith the skill or financial resources of smaller non-profit organisations.

Although based on a single case, the authors believe that the evidence how the use of innovative methodologies may be more appropriate to performing arts impact evaluations, even those less familiar to management researchers.

This paper offers insights into various methods of impact evaluation that may be of use to smaller non-profit arts organisations who may be constrained by limited skills and financial resources.

This paper provides an original contribution to understanding innovative methodologies to perform arts impact evaluations, particularly those assessing soft outcomes, and a contribution in recognising the role of academic researchers in performing such evaluations.