Search results

The purpose of this chapter is to discuss theory-based evaluation of public–private partnership (PPP) projects/programmes and to develop an intervention logical framework. It aims to draw attention to the need to go beyond the measurement of project/programme results to address not only the question of whether or not the project/programme worked but also the how and why questions. The study follows an interpretative methodology. It analytically discloses the mechanics of theory-based evaluation in relation to a ‘PPP theory’ and describes a theory-based analytical framework that portrays an explicit path towards ultimate impacts so as to assess, in a more systematic and integrated way, the success or failure of a PPP. Theory-based evaluation is a promising evaluation approach that would fit into the complexities of PPP projects/programmes and would expand the available toolbox of evaluators. Proper use of theory-based evaluation in PPP interventions contributes to better policy formulation and project implementation, thus leading to improved socio-economic benefits derived from PPP projects and programmes. The main contribution of the study is that it develops a ‘PPP theory’ and a related logical intervention framework drawing on theory-based approaches. Although the framework is developed for a representative sector, transport, it can easily be applied to any other PPP intervention.

In recent years, a wide range of psychosocial health interventions have been implemented among military service members and their families. However, there are questions over the evaluative rigor of these interventions. We conducted a systematic review of this literature, rating each relevant study (k = 111) on five evaluative rigor scales (type of control group, approach to participant assignment, outcome quality, number of measurement time points, and follow-up distality). The most frequently coded values on three of the five scales (control group type, participant assignment, and follow-up distality) were those indicating the lowest level of operationally defined rigor. Logistic regression results indicate that the evaluative rigor of intervention studies has largely remained consistent over time, with exceptions indicating that rigor has decreased. Analyses among seven military sub-populations indicate that interventions conducted among soldiers completing basic training, soldiers returning from combat deployment, and combat veterans have had, on average, the greatest evaluative rigor. However, variability in mean scores across evaluative rigor scales within sub-populations highlights the unique methodological hurdles common to different military settings. Recommendations for better standardizing the intervention evaluation process are discussed.

Purpose – Health system performance depends on production and use of quality health data and information. Routine health information systems (RHIS) are defined as systems that provide information at regular intervals of a year or less to meet predictable information needs. These include paper-based or electronic health records and facility- and district-level management information systems. RHIS are receiving increasing attention as an essential component of efficient, country-owned, integrated national systems. To guide investment decisions on RHIS, evidence is needed on which types of RHIS interventions work and which do not.

Design/methodology/approach – This chapter is a systematic review of the literature on the evaluation of RHIS interventions in low- and middle-income countries, starting from the premise that investments in RHIS could be better understood and so produce greater benefits than they currently do.

Findings – We describe the conceptual literature on the determinants of RHIS performance and its role in improving health systems functioning, discuss the evidence base on the effectiveness of strategies to improve RHIS performance, provide an overview of RHIS evaluation challenges, and make suggestions to improve the evidence base.

Originality/value – The goal is to help ensure that (a) RHIS interventions are appropriately designed and implemented to improve health systems functioning and (b) resulting RHIS information is used more effectively.

In the evaluation of most interventions in criminal justice settings, evaluators have no control over assignment to treatment and control/comparison conditions, which means that the treated and comparison groups may have differences that lead to biased conclusions regarding treatment effectiveness. Propensity score analysis can be used to balance the differences in the groups, which can be used in a number of ways to reduce biased conclusions regarding effectiveness. A review of propensity scoring studies was conducted for this chapter, where the limited number of evaluations of criminal justice interventions using these methods was identified. Due to the small number of these studies, research was also reviewed if propensity scoring had been employed to evaluate interventions that are similar to those in criminal justice systems. These studies are used as examples to demonstrate how the methods can be used to evaluate criminal justice interventions, the different ways propensity scores can be used to analyse treatment and comparison group differences, and the strengths and limitations of this approach. It is concluded that, while not appropriate for all interventions/settings, propensity score analysis can be useful in criminal justice arenas, at least to investigate the comparability of treatment and comparison groups, with suspected non-comparability being a common weakness of traditional quasi-experimental studies and frequently cited limitation in terms of drawing efficacy conclusions from such evaluations.

Assessment and identification of children with emotional and behavioral disorders (EBD) is complex and involves multiple techniques, levels, and participants. While federal law sets the general parameters for identification in school settings, these criteria are vague and may lead to inconsistencies in selection and interpretation of assessment measures. Assessment practice across school settings is greatly influenced by clinical guidelines such as the DSM-IV, which more specifically defines emotional and behavioral disorders and highlights the issue of co-morbidity. Before a student is assessed for special education eligibility under the IDEIA category of emotional disturbance, screening techniques and pre-referral interventions are needed. Positive Behavioral Supports and Response to Intervention models provide empirically supported frameworks for establishing the need for formal psychological assessment. Collaboration among members of the multidisciplinary team, including parents, helps to ensure that identification and intervention efforts have ecological validity. Tests and techniques vary considerably, but developmental histories, interviews, observations across settings, and behavioral checklists and rating scales are recommended, along with cognitive and achievement testing. While problems exist in the reliability and validity of projective techniques, they continue to be used in school-based assessment for EBD. Multitrait, multisetting, and multimethod approaches are essential for culturally fair assessment and reduction of bias in identification and placement.

The pressure on higher education providers (HEPs) and national programme partnerships to evaluate the impact of widening participation (WP) interventions has intensified as a result of wider changes in higher education (HE) policy and regulation, including the imposition of market forces. This chapter describes how policy stakeholder assumptions about how evaluation works and the outcomes it delivers have evolved over the last two decades. It shows how regulatory emphasis has shifted from a focus on monitoring and tracking, through to a call for return-on-investment analysis, before falling back on a pragmatic theory-informed approach. This chapter goes on to locate WP evaluation in the middle of a paradigm war, caught between proponents of a medicalised trial-based conception of evaluation methodology, and a practitioner-led position, which points to the complex contextual character of WP activities. It continues by exploring some of the many practical challenges faced by WP evaluators and argues that these have contributed to the sector’s perceived failure to deliver robust evidence of the impact of fair access activity. This chapter concludes with a look at the expanding market for WP evaluation products and services, which emerged in response to new flows of WP investment created by the 2012 increase in tuition fees.

In this paper, I offer a series of reflections on some issues involved in evaluating complex public policy programmes in a context of rapid change and against the background of an activist government setting exacting standards for the delivery of public service outcomes. I argue that the overlay of pressures on programme evaluation as a result of the engagement with such complex public policy interventions presents a series of challenges for evaluation.

A case study methodology was applied as a major component of a mixed-methods approach to the evaluation of a mobile dementia education and support service in the Bega Valley Shire, New South Wales, Australia. In-depth interviews with people with dementia (PWD), their carers, programme staff, family members and service providers and document analysis including analysis of client case notes and client database were used.

The strengths of the case study approach included: (i) simultaneous evaluation of programme process and worth, (ii) eliciting the theory of change and addressing the problem of attribution, (iii) demonstrating the impact of the programme on earlier steps identified along the causal pathway (iv) understanding the complexity of confounding factors, (v) eliciting the critical role of the social, cultural and political context, (vi) understanding the importance of influences contributing to differences in programme impact for different participants and (vii) providing insight into how programme participants experience the value of the programme including unintended benefits.

The broader case of the collective experience of dementia and as part of this experience, the impact of a mobile programme of support and education, in a predominately rural area grew from the investigation of the programme experience of ‘individual cases’ of carers and PWD. Investigation of living conditions, relationships, service interactions through observation and increased depth of interviews with service providers and family members would have provided valuable perspectives and thicker description of the case for increased understanding of the case and strength of the evaluation.

Identification, implementation, and ongoing evaluation of scientifically supported and effective-practice methods are fundamental and crucial elements of an effective educational program for all children and youth, including learners with Autism Spectrum Disorders (ASD). To be sure, there is an unambiguous link between use of interventions and supports that have empirically supported merit and positive school and post-school outcomes. This chapter examines progress in the wide-scale adoption of evidence-based methods with learners with ASD. We also offer recommendations for advancing effective-practice initiatives and discuss trends and themes connected to effective-practice use with students with ASD.

The roles and functions of a school psychologist are multifaceted. School psychologists are traditionally trained in areas of assessment, intervention, consultation, and program evaluation, though they often participate in prevention and crisis intervention efforts and program evaluation (Harvey & Struzziero, 2008). School psychologists work at district, building, and individual student levels to provide comprehensive and effective services to children and families. Despite a wide range of responsibilities, the school psychologist works in conjunction with other school professionals (e.g., general and special education teachers, speech-language pathologists, audiologists, social workers, principals) and parents to foster individual student success. This chapter presents the general roles and responsibilities of the school psychologist, as well as the school psychologist’s role within an interdisciplinary team when making appropriate educational decisions.