Search results

The purpose of this paper is to provide rationale, methodological guidance and clarity in the use of case study designs and theory driven approaches to evaluation of interventions to end homelessness.

Using an evaluation of a transitional shelter program aiming to support permanent exits from homelessness as an example, the authors show how case study designs and theory driven evaluation is well suited to the study of the effectiveness of homelessness interventions within the broader socio-political and economic context in which they are being implemented.

Taking account of the context as part of program evaluation and research on homelessness interventions moves away from blaming programs and individuals for systemic failures to better understanding of how the context influences successes and failures. Case study designs are particularly useful for studying implementation and the context which influences program outcomes. Theory driven evaluations and the use of realist evaluation as an approach can provide a broader understanding of how homelessness interventions work particularly for whom and under what conditions. These methodological and theoretical approaches provide a consistent strategy for evaluating programs aimed at ending homelessness.

There is a need for greater capacity in the homelessness sector to apply approaches to evaluation that take into account the broader socio-political and economic context in which programs are being implemented. Through the use of a case example, the authors provide guidance for application of case study design and theory driven approaches as a strategy for approaches programs aimed at ending homelessness.

The purpose of this chapter is to discuss theory-based evaluation of public–private partnership (PPP) projects/programmes and to develop an intervention logical framework. It aims to draw attention to the need to go beyond the measurement of project/programme results to address not only the question of whether or not the project/programme worked but also the how and why questions. The study follows an interpretative methodology. It analytically discloses the mechanics of theory-based evaluation in relation to a ‘PPP theory’ and describes a theory-based analytical framework that portrays an explicit path towards ultimate impacts so as to assess, in a more systematic and integrated way, the success or failure of a PPP. Theory-based evaluation is a promising evaluation approach that would fit into the complexities of PPP projects/programmes and would expand the available toolbox of evaluators. Proper use of theory-based evaluation in PPP interventions contributes to better policy formulation and project implementation, thus leading to improved socio-economic benefits derived from PPP projects and programmes. The main contribution of the study is that it develops a ‘PPP theory’ and a related logical intervention framework drawing on theory-based approaches. Although the framework is developed for a representative sector, transport, it can easily be applied to any other PPP intervention.

This chapter examines the nature and role of theory in criminal justice evaluation. A distinction between theories of and theories for evaluation is offered to clarify what is meant by ‘theory’ in the context of contemporary evaluation practice. Theories of evaluation provide a set of prescriptions and principles that can be used to guide the design, conduct and use of evaluation. Theories for evaluation include programme theory and the application of social science theory to understand how and why criminal justice interventions work to generate desired outcomes. The fundamental features of these three types of theory are discussed in detail, with a particular focus on demonstrating their combined value and utility for informing and improving the practice of criminal justice evaluation.

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.

A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.

Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.

The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.

The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.

As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.

Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.

An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.

A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.

Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.

This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.

The pressure on higher education providers (HEPs) and national programme partnerships to evaluate the impact of widening participation (WP) interventions has intensified as a result of wider changes in higher education (HE) policy and regulation, including the imposition of market forces. This chapter describes how policy stakeholder assumptions about how evaluation works and the outcomes it delivers have evolved over the last two decades. It shows how regulatory emphasis has shifted from a focus on monitoring and tracking, through to a call for return-on-investment analysis, before falling back on a pragmatic theory-informed approach. This chapter goes on to locate WP evaluation in the middle of a paradigm war, caught between proponents of a medicalised trial-based conception of evaluation methodology, and a practitioner-led position, which points to the complex contextual character of WP activities. It continues by exploring some of the many practical challenges faced by WP evaluators and argues that these have contributed to the sector’s perceived failure to deliver robust evidence of the impact of fair access activity. This chapter concludes with a look at the expanding market for WP evaluation products and services, which emerged in response to new flows of WP investment created by the 2012 increase in tuition fees.

The purpose of this paper is to discuss different approaches to evaluation taken in the case of two regeneration programmes in England and Wales and identify how assumptions about the nature of policy and programme implementation affect evaluation and considerations of its use in the public sector.

Using an analysis of evaluation reports relating to two regeneration programmes, a distinction is drawn between two approaches to evaluation. This distinction is then articulated further with reference to differing perspectives in policy implementation studies. Related issues of policy and programme success and failure are discussed, in addition to the potential for policy learning.

There is a difference between the two approaches to evaluation employed in the two cases, which is likely to reflect contrasting beliefs about the nature and process of implementation. Implementation research indicates that while a process of evaluating progress against a set of specific objectives clearly has legitimacy in a public sector environment, it is likely to be inconclusive or even counterproductive. This is due to the inevitability of policy evolution, the multiple reformations of policy at the micro level, and the constraints engendered by norms of appropriacy in the wider policy environment.

This article suggests how ongoing debates in implementation studies can inform and illuminate decisions made about approaches to evaluation. The dynamic between conceptions of implementation and the purpose of evaluation is illustrated through a comparison of the use of evaluation in two regeneration programmes.

Public sector interest in methods for priority setting and program or policy evaluation has grown considerably over the last several decades, given increased expectations for accountable and efficient use of resources and emphasis on evidence‐based decision making as a component of good management practice. While there has been some occasional effort to conduct evaluation of priority setting projects, the literatures around priority setting and evaluation have largely evolved separately. In this paper, the aim is to bring them together.

The contention is that evaluation theory is a means by which evaluators reflect upon what it is they are doing when they do evaluation work. Theories help to organize thinking, sort out relevant from irrelevant information, provide transparent grounds for particular implementation choices, and can help resolve problematic issues which may arise in the conduct of an evaluation project.

A detailed review of three major branches of evaluation theory – methods, utilization, and valuing – identifies how such theories can guide the development of efforts to evaluate priority setting and resource allocation initiatives. Evaluation theories differ in terms of their guiding question, anticipated setting or context, evaluation foci, perspective from which benefits are calculated, and typical methods endorsed.

Choosing a particular theoretical approach will structure the way in which any priority setting process is evaluated. The paper suggests that explicitly considering evaluation theory makes key aspects of the evaluation process more visible to all stakeholders, and can assist in the design of effective evaluation of priority setting processes; this should iteratively serve to improve the understanding of priority setting practices themselves.