Search results

This paper aims to present the process evaluation for a two‐year (2008‐2010) participatory action research project focusing on home‐school partnership in health learning, undertaken within the Schools for Health in Europe (SHE) in Eastern Finland.

Two intervention schools and two control schools (grade 5 pupils, parents, and selected school personnel) participated in a study. Process evaluation data were collected from intervention schools after 10 months of participation, by interviewing two classroom teachers and three families. In addition, program documents and relevant statistics were collected from schools during the intervention.

Teachers' opinions on the development process varied from more concrete expectations (School A teacher) to overall satisfaction to implementation (School B teacher). Parents believed that their children would benefit from the project later in life. The context and differences of the school environments were likely to affect the development process at the school level.

This paper demonstrates a process evaluation in two schools and, therefore, limits the generalizability of the findings.

The process evaluation was an essential part of this intervention study and may provide a useful structure and an example for process evaluation for future school‐based health intervention studies.

This study highlights the importance of planning the process evaluation structure before the start of the intervention, brings out the relevance of systematically assessing the process while it is ongoing, and illustrates process evaluation in an action research project.

This study aims to examine the extent to which “What Works” reviews in youth justice enable understanding of the features of effectiveness (what works, for whom, in what circumstances and why?) specified in the Effects–Mechanisms–Moderators–Implementation–Economic cost (EMMIE) framework.

The EMMIE framework examined findings within a sample of “What Works” style reviews of preventative youth justice intervention effectiveness.

“What Works” style reviews of evaluations of preventative youth justice interventions often omit the requisite details required to examine all of the necessary elements of effectiveness contained within the EMMIE framework. While effectiveness measures were typically provided, the dominant evaluation evidence-base struggles to consider moderators of effect, mechanisms of change, implementation differences and cost-effectiveness. Therefore, “What Works” samples cannot facilitate sufficient understanding of “what works for whom, in what circumstances and why?”. The authors argue that Realist Synthesis can fill this gap and shed light on the contexts that shape the mechanisms through which youth justice interventions work.

The authors extended the approach adopted by an earlier review of effectiveness reviews (Tompson et al., 2020), considering more recent reviews of the effectiveness of preventative interventions using the EMMIE framework. Unlike previous reviews, the authors prioritised the utility of the EMMIE framework for assessing the factors affecting the effectiveness of preventative interventions in youth justice.

Models of integrated care are prime examples of complex interventions, incorporating multiple interacting components that work through varying mechanisms to impact numerous outcomes. The purpose of this paper is to explore summative, process and developmental approaches to evaluating complex interventions to determine how to best test this mess.

This viewpoint draws on the evaluation and complex intervention literatures to describe the advantages and disadvantages of different methods. The evaluation of the electronic patient reported outcomes (ePRO) mobile application and portal system is presented as an example of how to evaluate complex interventions with critical lessons learned from this ongoing study.

Although favored in the literature, summative and process evaluations rest on two problematic assumptions: it is possible to clearly identify stable mechanisms of action; and intervention fidelity can be maximized in order to control for contextual influences. Complex interventions continually adapt to local contexts, making stability and fidelity unlikely. Developmental evaluation, which is more conceptually aligned with service-design thinking, moves beyond these assumptions, emphasizing supportive adaptation to ensure meaningful adoption.

Blended approaches that incorporate service-design thinking and rely more heavily on developmental strategies are essential for complex interventions. To maximize the benefit of this approach, three guiding principles are suggested: stress pragmatism over stringency; adopt an implementation lens; and use multi-disciplinary teams to run studies.

This viewpoint offers novel thinking on the debate around appropriate evaluation methodologies to be applied to complex interventions like models of integrated care.

The purpose of this chapter is to discuss theory-based evaluation of public–private partnership (PPP) projects/programmes and to develop an intervention logical framework. It aims to draw attention to the need to go beyond the measurement of project/programme results to address not only the question of whether or not the project/programme worked but also the how and why questions. The study follows an interpretative methodology. It analytically discloses the mechanics of theory-based evaluation in relation to a ‘PPP theory’ and describes a theory-based analytical framework that portrays an explicit path towards ultimate impacts so as to assess, in a more systematic and integrated way, the success or failure of a PPP. Theory-based evaluation is a promising evaluation approach that would fit into the complexities of PPP projects/programmes and would expand the available toolbox of evaluators. Proper use of theory-based evaluation in PPP interventions contributes to better policy formulation and project implementation, thus leading to improved socio-economic benefits derived from PPP projects and programmes. The main contribution of the study is that it develops a ‘PPP theory’ and a related logical intervention framework drawing on theory-based approaches. Although the framework is developed for a representative sector, transport, it can easily be applied to any other PPP intervention.

It has become accepted practice to include an evaluation alongside learning programmes that take place at work, as a means of judging their effectiveness. There is a tendency to focus such evaluations on the relevance of the intervention and the amount of learning achieved by the individual. The aim of this review is to examine existing evaluation frameworks that have been used to evaluate education interventions and, in particular, assess how these have been used and the outcomes of such activity.

A scoping review using Arskey and O’Malley’s five stage framework was undertaken to examine existing evaluation frameworks claiming to evaluate education interventions.

Forty five articles were included in the review. A majority of papers concentrate on learner satisfaction and/or learning achieved. Rarely is a structured framework mentioned, or detail of the approach to analysis cited. Typically, evaluations lacked baseline data, control groups, longitudinal observations and contextual awareness.

This review has implications for those involved in designing and evaluating work-related education programmes, as it identifies areas where evaluations need to be strengthened and recommends how existing frameworks can be combined to improve how evaluations are conducted.

This scoping review is novel in its assessment and critique of evaluation frameworks employed to evaluate work-related education programmes.

In recent years, a wide range of psychosocial health interventions have been implemented among military service members and their families. However, there are questions over the evaluative rigor of these interventions. We conducted a systematic review of this literature, rating each relevant study (k = 111) on five evaluative rigor scales (type of control group, approach to participant assignment, outcome quality, number of measurement time points, and follow-up distality). The most frequently coded values on three of the five scales (control group type, participant assignment, and follow-up distality) were those indicating the lowest level of operationally defined rigor. Logistic regression results indicate that the evaluative rigor of intervention studies has largely remained consistent over time, with exceptions indicating that rigor has decreased. Analyses among seven military sub-populations indicate that interventions conducted among soldiers completing basic training, soldiers returning from combat deployment, and combat veterans have had, on average, the greatest evaluative rigor. However, variability in mean scores across evaluative rigor scales within sub-populations highlights the unique methodological hurdles common to different military settings. Recommendations for better standardizing the intervention evaluation process are discussed.

Football exercise as an intervention for people with severe mental health problems has seen an increasing interest in the past years. To date, there is, however, no comprehensive review of the empirical evidence regarding the effectiveness of these interventions. In this review, the authors have comprised the research findings from the peer-review literature as well as the theoretical approaches to football exercise as an adjunct treatment. This overview will be informative to everybody who is planning to develop a football intervention for this population as well as to the people who are preparing evaluation studies that measure the effectiveness of such interventions. The paper aims to discuss these issues.

The authors identified research papers in the peer-review literature that feature empirical findings on “football interventions” that aim at improving mental and/or physical well-being in participants with mental health problems. The authors are using the term “football intervention” here in the sense that the participants actively took part in football exercise, so the authors excluded studies in which the participants only watched football or used football as a metaphor to discuss mental health problems. In a table, the authors indicate the definition of the target group, targeted outcomes, measured outcomes, form and frequency of the intervention as well as the research method(s).

The authors identified 16 studies on 15 projects. The majority of studies were qualitative and had positive findings in which the participants reported increased well-being and connectedness, elevation of symptoms and improved physical well-being. The outcomes of the quantitative studies, however, were mixed with some results suggesting that not all intended goals were achieved. There seems to be a need for more quantitative studies to triangulate the qualitative findings. Interestingly, most interventions take place in the UK. Many studies fail to give detailed methodological information and often the aims of the interventions are vague or not stated at all.

Due to the heterogeneity of the studies and relative scarcity of evaluation projects on football interventions for people with mental health problems, the authors could not conduct an in-depth systematic review. Furthermore, the information on methods was often unsatisfying and despite efforts to get more detailed input from the authors of cited papers, those gaps could not always be filled. Instead of coming up with a crystal-clear summary of whether and how football interventions work for everybody, topics were identified that need to be addressed in the planning of interventions, in evaluation studies, in implementation efforts and in the theoretical discourse.

This paper constitutes a helpful overview for everybody who is interested in the theoretical background of football interventions for people with mental health problems, for people who are planning to develop respective interventions, for researchers who engage in evaluation projects that look into the effectiveness of football interventions (or similar exercise interventions) as well as for the people who are interested in how football interventions can be implemented. This paper is likely to make a contribution to the advancement of alternative exercise interventions that aim at improving mental, physical and social health in people with mental health problems.

This paper will help putting the topic of football interventions (and similar, alternative exercise interventions) further up on the public health agenda by providing an overview of the empirical evidence at hand and by specifying advantages of the approach as well as pointing out actions that need to be taken to make football a recognised, evidence based and viable option for adjunct mental health treatment that is attractive to potential participants as well as funders as well as to the potential participants.

There is no comprehensive summary to date that provides a (reasonably) systematic overview of empirical findings for football interventions for people with MH problems. Furthermore, the literature on the theoretical background of these interventions has been somewhat patchy and heterogonous. This paper aims at filling both these gaps and identifies the issues that need to be covered in the planning of respective interventions and evaluations. This paper will be useful to everybody who is developing football interventions (or similar alternative adjunct exercise interventions), who is conducting evaluation research in this area and who is interested in the implementation of football interventions.

This paper reports on the implementation and evaluation of the JOBS programme in Ireland. This is a training intervention to promote re‐employment and improve mental health among unemployed people that was implemented on a pilot basis in the border region of the Republic and Northern Ireland. Programme participants were unemployed people recruited from local training and employment offices and health agencies. The evaluation indicated that the programme was implemented successfully and led to improved psychological and re‐employment outcomes for the intervention group, lasting up to 12 months post‐intervention. This paper reflects on the implementation issues that arose in adapting an international evidence‐based programme to the local setting and considers the implications of the evaluation findings for the roll out of the programme on a larger scale.

The purpose of this paper is to examine whether a successful implementation of an intervention could result in an effect evaluated independently from a process evaluation. It was achieved by evaluating the effects of an intervention, the “employeeship program,” designed to strengthen the psychosocial work environment through raising employees’ awareness and competence in interpersonal relationships and increasing their responsibility for their everyday work and working environment.

An employeeship intervention program was developed to improve the psychosocial work environment through reducing conflict among employees and strengthening the social community, empowering leadership, and increasing trust in management. An earlier process evaluation of the program found that it had been implemented successfully. The present effect evaluation supplemented this by examining its effect on the psychosocial work environment using two waves of the organization’s internal survey and comparing changes in the intervention unit at two points and against the rest of the organization.

The intervention was effective in improving the psychosocial work environment through reducing conflicts among employees and strengthening the social community, empowering leadership, and increasing trust in management.

More attention should be paid to developing and increasing positive psychosocial experiences while simultaneously reducing negative psychosocial experiences, as this employeeship intervention demonstrated.

An intervention focusing on employeeship is an effective way to achieve a healthier psychosocial work environment with demonstrable benefits for individuals and the working unit.

Although organizational-level interventions are complex processes, evaluations that focus on process and effect can offer insights into the workings of successful interventions.

Realist evaluation seeks to answer the question of “what works for whom in which circumstances?” through developing and testing middle range theories (MRTs). MRTs are programme theories that outline how certain mechanisms of an intervention work in a specific context to bring about certain outcomes. In this paper, the authors tested an initial MRT about the mechanism of participation. The authors used evidence from a participatory organisational intervention in five worksites of a large multi-national organisation in the US food service industry.

Qualitative data from 89 process tracking documents and 24 post-intervention, semi-structured interviews with intervention stakeholders were analysed using template analysis.

The operationalised mechanism was partial worksite managers’ engagement with the research team. Six contextual factors (e.g. high workload) impaired participation, and one contextual factor (i.e. existing participatory practices) facilitated participation. Worksite managers’ participation resulted in limited improvement in their awareness of how working conditions can impact on their employees’ safety, health, and well-being. Based on these findings, the authors modified the initial MRT into an empirical MRT.

This paper contributes to the understanding of “what works for whom in which circumstances” regarding participation in organisational interventions.