Search results

The purpose of this chapter is to discuss theory-based evaluation of public–private partnership (PPP) projects/programmes and to develop an intervention logical framework. It aims to draw attention to the need to go beyond the measurement of project/programme results to address not only the question of whether or not the project/programme worked but also the how and why questions. The study follows an interpretative methodology. It analytically discloses the mechanics of theory-based evaluation in relation to a ‘PPP theory’ and describes a theory-based analytical framework that portrays an explicit path towards ultimate impacts so as to assess, in a more systematic and integrated way, the success or failure of a PPP. Theory-based evaluation is a promising evaluation approach that would fit into the complexities of PPP projects/programmes and would expand the available toolbox of evaluators. Proper use of theory-based evaluation in PPP interventions contributes to better policy formulation and project implementation, thus leading to improved socio-economic benefits derived from PPP projects and programmes. The main contribution of the study is that it develops a ‘PPP theory’ and a related logical intervention framework drawing on theory-based approaches. Although the framework is developed for a representative sector, transport, it can easily be applied to any other PPP intervention.

This chapter examines the nature and role of theory in criminal justice evaluation. A distinction between theories of and theories for evaluation is offered to clarify what is meant by ‘theory’ in the context of contemporary evaluation practice. Theories of evaluation provide a set of prescriptions and principles that can be used to guide the design, conduct and use of evaluation. Theories for evaluation include programme theory and the application of social science theory to understand how and why criminal justice interventions work to generate desired outcomes. The fundamental features of these three types of theory are discussed in detail, with a particular focus on demonstrating their combined value and utility for informing and improving the practice of criminal justice evaluation.

This paper aims to explore the evaluation theory in a field closely related to corporate communication and public relations (PR) as well as in other disciplines and argues that embracing the evaluation theory more broadly can break the “stasis” and “deadlock” identified in evaluation of corporate communication and PR. Specifically, this analysis seeks to show that a transdisciplinary approach can contribute to standards and demonstration of impact – two long-sought goals in evaluation of corporate communication and PR – as well as inform methodology.

Critical analysis is applied to review the evaluation theory in a number of fields, including international development, public administration, management and health communication, compared with major frameworks, models and methods used for evaluation of corporate communication and PR.

This analysis shows that the evaluation theory in other fields and related theory of change, program theory and program logic models can contribute to advancing evaluation of corporate communication and PR in three ways: identifying standards in terminology and approaches, shifting focus from activities and outputs to outcomes and impact and applying appropriate and rigorous methodology.

While this paper does not present new empirical data, it expands the theoretical perspectives, models and methods applied to the evaluation of corporate communication and PR and identifies new directions for research.

As well as expanding the evaluation theory and opening up new ground for research, this analysis identifies a need for structural change in the field of practice.

A meta-evaluation is an assessment of evaluation practices. Meta-evaluations include assessments of validity and usefulness of two or more studies that focus on the same issues. Every performance audit is grounded explicitly or implicitly in one or more theories of program evaluation. A deep understanding of alternative theories of program evaluation is helpful to gain clarity about sound auditing practices. We present a review of several theories of program evaluation.

This study includes a meta-evaluation of seven government audits on the efficiency and effectiveness of tourism departments and programs. The seven tourism-marketing performance audits are program evaluations for: Missouri, North Carolina, Tennessee, Minnesota, Australia, and two for Hawaii. The majority of these audits are negative performance assessments. Similarly, although these audits are more useful than none at all, the central conclusion of the meta-evaluation is that most of these audit reports are inadequate assessments. These audits are too limited in the issues examined; not sufficiently grounded in relevant evaluation theory and practice; and fail to include recommendations, that if implemented, would result in substantial increases in performance.

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.

A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.

Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.

The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.

The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.

As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.

Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.

The chapter will explore a collaborative theory of change approach that the authors used to evaluate three projects. The three projects worked with young people out of school in different ways to enable the young people to become agents of change in tackling the causes of alcohol misuse in their local Scottish communities. A theory of change approach provides a way of conceptualising programmes from inception, through to implementation and the evaluation of outcomes, in order to develop an understanding of how they work, for whom and in what circumstances. Using a collaborative model of this approach challenged prevailing notions of evaluation being the job of the evaluator and situated evaluation as a shared endeavour with the project staff. We outline the key attributes of such a collaborative model of theory of change and reflect on how this model can contribute to the evaluation of out-of-school activity.

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.

An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.

A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.

Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.

This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.

Public sector interest in methods for priority setting and program or policy evaluation has grown considerably over the last several decades, given increased expectations for accountable and efficient use of resources and emphasis on evidence‐based decision making as a component of good management practice. While there has been some occasional effort to conduct evaluation of priority setting projects, the literatures around priority setting and evaluation have largely evolved separately. In this paper, the aim is to bring them together.

The contention is that evaluation theory is a means by which evaluators reflect upon what it is they are doing when they do evaluation work. Theories help to organize thinking, sort out relevant from irrelevant information, provide transparent grounds for particular implementation choices, and can help resolve problematic issues which may arise in the conduct of an evaluation project.

A detailed review of three major branches of evaluation theory – methods, utilization, and valuing – identifies how such theories can guide the development of efforts to evaluate priority setting and resource allocation initiatives. Evaluation theories differ in terms of their guiding question, anticipated setting or context, evaluation foci, perspective from which benefits are calculated, and typical methods endorsed.

Choosing a particular theoretical approach will structure the way in which any priority setting process is evaluated. The paper suggests that explicitly considering evaluation theory makes key aspects of the evaluation process more visible to all stakeholders, and can assist in the design of effective evaluation of priority setting processes; this should iteratively serve to improve the understanding of priority setting practices themselves.

This paper considers the implications of mass communications theory on public relations (PR) evaluation and briefly reviews mass communication effects, persuasion, and cognition, attitude and behaviour change theories. The implications for evaluation are then examined. Reliance on domino models is shown to be too simplistic. It is suggested that claims of PR behavioural effects may be unrealistic and it is argued that more moderate and/or alternative goals are needed if preordained failure is to be avoided. Evaluation results must be interpreted cautiously so that further significance that is not supported by theory is not assumed. This paper shows how the concept of PR evaluation could be widened to include formative evaluation and broad environmental monitoring, which are especially important in identifying and understanding why and how communication works, what its effects are, what factors restrict or facilitate effectiveness and under what conditions success can be maximised.