Perspectives on Evaluating Criminal Justice and Corrections: Volume 13

Questions regarding the effectiveness of criminal justice efforts to reduce crime have dominated social and political thinking in this area for more than a century (Bowen, 2011). During this time a number of philosophical shifts regarding the aims of correctional systems have occurred, fuelled typically by the prevailing political standpoint (McGuire, 2005). At the start of the twentieth century, policymakers in the United States and United Kingdom placed faith in the rehabilitative ideal and offender ‘treatment’-dominated corrections policies (Cullen & Gendreau, 2000). In this context ‘treatment’ refers to a range of interventions designed to alter the individual, contextual and social factors that sustain offending behaviour (Hollin, 1999). This remained the prevailing perspective for the subsequent seven decades until questions arose regarding the quality of ‘state run’ corrections facilities in the United States in the early 1970s. At this point, evaluation science was one of many influences on a change of policy (Hollin, 1999). Martinson's (1974) now infamous research synthesis, described by Glaes (1998, p. 713) as ‘a watershed moment’, provided politicians and policymakers with greater justification for changing the focus of corrections policy. Although arguably it was observers misreporting of Martinson's claims about the evidence which were most influential, rather than the review itself. According to Martinson (1974):It is just possible that some of our treatment programs are working to some extent, but that our research is so bad that it is incapable of telling. Having entered this very serious caveat, I am bound to say that these data … give us very little reason to hope that we have in fact found a sure way of reducing recidivism through rehabilitation. This is not to say that we found no instances of success or partial success; it is only to say that these instances have been isolated, producing no clear pattern to indicate the efficacy of any particular method of treatment. (p. 49)

This chapter examines the nature and role of theory in criminal justice evaluation. A distinction between theories of and theories for evaluation is offered to clarify what is meant by ‘theory’ in the context of contemporary evaluation practice. Theories of evaluation provide a set of prescriptions and principles that can be used to guide the design, conduct and use of evaluation. Theories for evaluation include programme theory and the application of social science theory to understand how and why criminal justice interventions work to generate desired outcomes. The fundamental features of these three types of theory are discussed in detail, with a particular focus on demonstrating their combined value and utility for informing and improving the practice of criminal justice evaluation.

This chapter describes Andrews and Bonta's (2006) Principles of Effective Offender Treatment and its relevance for the treatment of sexual offenders. The three principles of this model are Risk, Needs and Responsivity. Each of these is described in some detail with the greatest emphasis being placed on general responsivity which is one of the two parts of the Responsivity Principle. Our interpretation of general responsivity differs from the view of others (e.g. Hanson et al., 2009) who define this aspect of Responsivity in terms of Cognitive Behaviour Therapy (CBT). While Andrews and Bonta indicate that within their meta-analyses, CBT programmes were the ones most likely to succeed; such programmes were not at all effective. It seems to us that a far more important aspect of general responsivity is what Andrews and Bonta describe as the Core Correctional Practices (CCP) which have to do with the way in which treatment is delivered. We review the CCPs in some detail and provide other evidence indicating that the style of treatment delivery is the crucial factor in determining effectiveness.

The chapter considers the change of position of the Home Office on the value of randomised controlled trials (RCTs) in England and Wales which took place around 2003 after the end of the Crime Reduction Programme (CRP). Before the CRP Home Office researchers had shown little interest in RCTs; after it, they came close to arguing that no other kinds of evaluation research were worth doing. This represented a reversal of a position that had dominated Home Office thinking on the issue for almost 30 years – that RCTs were in general impractical and unlikely to produce clear-cut results. This view was based in part on the experience of RCTs in the 1970s, which led influential researchers to conclude that the method could not be transferred from medicine to criminal justice. But, disappointed with the lack of definite results from the CRP, the Home Office turned back to RCTs as a potential source of certainty about what works. The chapter considers two recent scholarly exchanges on the question, in relation to an evaluation of a community crime reduction programme, for which an experimental design was attempted but not achieved, and to Lawrence Sherman's recent advocacy of RCTs and his use of research on restorative justice as an example of the successful use of the method. The chapter argues that the restorative justice research, while of very high quality, does not provide as clear an example of the use of an RCT as Sherman claims, and concludes with some reflections on the inherent difficulties of criminal justice evaluation, and on the lack of a predictable, rational relationship between research quality and policy influence.

In this chapter, we first describe the all governing principles of treatment for sexual offenders that maximise effectiveness. These are derived from Andrews and Bonta's (2006) summary of a variety of meta-analyses of outcome studies. From this source and others, we then claim that there are three elements essential to effective treatment: (1) targeting criminogenic features; (2) employing empirically sound procedures to modify these targets; and (3) delivering treatment in an effective psychotherapeutic way. Next we describe our treatment approach that emphasises these crucial elements within a strength-based programme that is motivational and provides Ward's (2002) Good Lives Model as the framework. We then challenge the broadly accepted idea that the Random Controlled Trial (RCT) is the only basis upon which inferences about treatment effectiveness can be derived. We point to methodological, practical and ethical problems associated with implementing an RCT study and offer at least two alternatives: the so-called ‘incidental design’ which compares the outcome of the treated group with a matched (but not randomly assigned) group from the same or similar setting to the treated group; and a strategy where the recidivism rate of treated group is compared with what would be expected on the basis of risk assessments of each of the treated subjects.

The chapter considers the ethical problems engendered by random assignment and privacy concerns in randomised controlled experiments and cluster randomised trials. The particular focus is on procedural, legislative and technical approaches to reducing or avoiding the problems. Examples are given from a variety of disciplines including health and education, though the main emphasis is on research in crime and delinquency.

There has been an increasing realisation within government circles that gaining insight into how and why an intervention works or does not is as important as measuring any change that it brings about. Without understanding the mechanisms for change, ensuring that the intervention is effective when transposed to different contexts can be highly challenging. However, while the need for high-quality, robust qualitative research is recognised, the theoretical and methodological tools available to researchers within the field have not kept up. Government-commissioned evaluation requires a methodology that provides genuine insight into how policy and interventions work on the ground and findings that can be generalised beyond the specific samples upon which they are based. In order to fill this conceptual gap, NatCen Social Research has developed an approach to qualitative research that draws on a wide range of existing traditions but that is robust and coherent enough to meet the needs of CJS evaluations. Recent work has led to a new articulation of this approach and a new moniker, ‘critical qualitative theory’ or CQT. This chapter describes the key tenets of CQT and draws on three of published studies of interventions that are focused on or relate to the criminal justice system to illustrate the methodological advantages the approach brings. In particular, the chapter explores the different approaches taken by the authors to the issues of generalisability, the balance between inductive and deductive procedures and how they have approached analysing the data at a case and a theme level. The implication of the different theoretical and methodological choices made by the authors is then discussed in terms of the nature and quality of the authors’ analysis.

Traditionally evaluators of offending behaviour programmes have examined group-level mean change in treatment targets without acknowledging the potential variability of change at an individual level. Clinically significant change, although used widely in the therapy literature generally, has only recently been examined within forensic therapeutic contexts. This chapter provides an overview of key concepts, and the published literature in which clinically significant change has been examined within forensic samples is reviewed. It is concluded that although this technique has the potential to validate programme theory, it is yet to be used to its full potential within a forensic context.

In the evaluation of most interventions in criminal justice settings, evaluators have no control over assignment to treatment and control/comparison conditions, which means that the treated and comparison groups may have differences that lead to biased conclusions regarding treatment effectiveness. Propensity score analysis can be used to balance the differences in the groups, which can be used in a number of ways to reduce biased conclusions regarding effectiveness. A review of propensity scoring studies was conducted for this chapter, where the limited number of evaluations of criminal justice interventions using these methods was identified. Due to the small number of these studies, research was also reviewed if propensity scoring had been employed to evaluate interventions that are similar to those in criminal justice systems. These studies are used as examples to demonstrate how the methods can be used to evaluate criminal justice interventions, the different ways propensity scores can be used to analyse treatment and comparison group differences, and the strengths and limitations of this approach. It is concluded that, while not appropriate for all interventions/settings, propensity score analysis can be useful in criminal justice arenas, at least to investigate the comparability of treatment and comparison groups, with suspected non-comparability being a common weakness of traditional quasi-experimental studies and frequently cited limitation in terms of drawing efficacy conclusions from such evaluations.

The Offender Engagement Programme (OEP) within National Offender Management Service (NOMS) has been built on our collaborative approach with probation trusts to develop approaches which can support more effective offender engagement. Drawing areas of potential practice development from our early fieldwork and literature review, events for trusts and reference groups with practitioners and middle managers, we identified where more focussed work with trusts would develop learning and evidence which could be shared. In response to an invitation to express an interest, 22 trusts committed themselves to taking forward an OEP pilot, starting between March and May 2011. External evaluation is planned for nine trusts in total and is being undertaken by our research partners: a team from the Institute of Criminal Policy Research (ICPR) Birkbeck College London and Leicester University, and a team based at Sheffield University which includes Fergus McNeill from Glasgow University.

This chapter looks at how we implemented pilot work on approaches to involving offenders in Sentence Planning; Developing Effective Engagement Skills through training and continuous professional development; and a model for Reflective Supervision by senior and middle managers to support effective engagement. It also examines how the methodology for the external and internal evaluation was developed, and what we are hoping to get out of the evaluation. In short, the purpose of this work is to investigate and test the core proposition that the relationship between the offender and the practitioner can be a powerful means of changing behaviour.

This chapter looks at the role qualitative evaluation can play in the external review of the Probation Service, the development of an evaluation framework for ongoing assessment and how it can be used to develop new elements of the service. How is this different from the use of existing data by the service?

This takes us to the kind of information that is used by the probation service, to make judgements about the effectiveness of its programmes and the impact on offenders. The main contention is that this is in the main quantitative data, and reports on levels of re-offending. The data is standardised so that it can be used to make comparisons between different types of sentence and criminal justice intervention.

Our contention is that this information is limited in what it says for two main reasons. Firstly, quantitative data tends to report on impact, that is whether an offender has committed another crime after engagement with the probation service, or whether there are patterns of behaviour between offences, individual circumstances and the likelihood or re-offending. In short, the data is a snapshot of whether an offender has changed his or her behaviour or not. However, this chapter will illustrate how quantitative data misses an understanding of how behaviour changes and why behaviour does not change. As a result, this leaves the service with a limited understanding of how it is working.

Secondly, the chapter will argue that the existing probation framework itself creates quite specific definitions of which data is relevant and which data is not relevant. We will give examples of narratives that offenders offer in group sessions that provide rich material about their lives, pressures and offence. But evidence suggests that this information is not used to inform programmes, re-assess and review an offender's own progression towards re-offending or a life without crime.

When research is complete and findings concluded, is research really just about publishing findings? Have you ever wondered whether your research will have any real impact in the world? In our opinion, we believe a bigger obstacle awaits us at the end of a research project. This being the implementation of research findings. We share with you our combined experience spanning over three decades of research – the challenges and learnings we have gathered along the way as we aim to translate research findings into real world, everyday practical reality. Our research experience is generally about human behaviour and psychology within the forensic population. As a result, the implementation of findings is crucial to inform policy makers, practitioners, local community agencies and government departments about the way to work with this population. In this chapter we provide a practical step-by-step approach to how we aim to implement our findings in research. We look at the problem of linking between theory and practice alongside what is currently in practice today. We discuss the challenges we have faced and what we have done to ‘Making real-world research stick at the coal face’. We wish you all the best with your research and hope that you find it as challenging and rewarding as we have!

Brad Astbury is research fellow in the Centre for Program Evaluation, Melbourne Graduate School of Education at the University of Melbourne where he lectures within the Masters of Evaluation course. His interests lie in evaluation theory and social research methodology. Brad has conducted evaluations in a number of areas, including corrections, education, health promotion and various family and community service interventions.