Search results

Calls for the development and dissemination of evidence-based programs to support children and families have been increasing for decades, but progress has been slow. This paper aims to argue that a singular focus on evaluation has limited the ways in which science and research is incorporated into program development, and advocate instead for the use of a new concept, “scientific accompaniment,” to expand and guide program development and testing.

A heuristic is provided to guide research–practice teams in assessing the program’s developmental stage and level of evidence.

In an idealized pathway, scientific accompaniment begins early in program development, with ongoing input from both practitioners and researchers, resulting in programs that are both effective and scalable. The heuristic also provides guidance for how to “catch up” on evidence when program development and science utilization are out of sync.

While implementation models provide ideas on improving the use of evidence-based practices, social service programs suffer from a significant lack of research and evaluation. Evaluation resources are typically not used by social service program developers and collaboration with researchers happens late in program development, if at all. There are few resources or models that encourage and guide the use of science and evaluation across program development.

A well-funded, four-year integrated care programme was implemented in south London. The programme attempted to integrate care across primary, acute, community, mental health and social care. The purpose of this paper is to reduce hospital admissions and nursing home placements. Programme evaluation aimed to identify what worked well and what did not; lessons learnt; the value of integrated care investment.

Qualitative data were obtained from documentary analysis, stakeholder interviews, focus groups and observational data from programme meetings. Framework analysis was applied to stakeholder interview and focus group data in order to generate themes.

The integrated care project had not delivered expected radical reductions in hospital or nursing home utilisation. In response, the scheme was reformulated to focus on feasible service integration. Other benefits emerged, particularly system transformation. Nine themes emerged: shared vision/case for change; interventions; leadership; relationships; organisational structures and governance; citizens and patients; evaluation and monitoring; macro level. Each theme was interpreted in terms of “successes”, “challenges” and “lessons learnt”.

Evaluation was hampered by lack of a clear evaluation strategy from programme inception to conclusion, and of the evidence required to corroborate claims of benefit.

Key lessons learnt included: importance of strong clinical leadership, shared ownership and inbuilt evaluation.

Primary care was a key player in the integrated care programme. Initial resistance delayed implementation and related to concerns about vertical integration and scepticism about unrealistic goals. A focus on clinical care and shared ownership contributed to eventual system transformation.

With the recent release of the National Standard for Mental Health and Well-Being for Post-Secondary Students, there is increased interest to integrate research and practice for mental health services on post-secondary campuses. Participant-oriented research is a useful framework to bridge this gap. This paper aims to describe the program development and evaluation process and reports challenges and lessons learned to inform future implementation strategies for similar endeavours.

A participant-oriented research approach was used to revise and evaluate an innovative interdisciplinary resilience program, entitled Roots of Resiliency, for post-secondary students.

This case analysis used the development and evaluation of Roots of Resiliency to demonstrate some of the strategies and challenges that exist for participant-oriented research related to mental health in the post-secondary context. Collaborative relationships among the various development team members contributed to an overall positive experience. Some challenges that others who work in post-secondary mental health field may consider include the need for content expertise, the ongoing need for communication among team members and the need for an effective system to give voice to all participants.

Any mental health program has a cultural component and is best co-developed by the particular students (e.g. indigenous students) who are to be served by the program. In this regard, the co-design and shared development and evaluation of the current mental health program is an example that can be emulated in other programs within the post-secondary context.

This paper aims to explore the learning needs of general practitioners (GPs) involved in commissioning mental health provision in England, and offer an evaluation of a leadership and commissioning skills development programme for Mental Health Commissioners.

Retrospective mixed method, including online mixed method survey, rating participants’ knowledge, skills, abilities, semi-structured telephone interviews and third-party questionnaires were used. Results were analysed for significant differences using the Wilcoxon Signed Ranks test. Open-ended responses and interview transcripts were analysed thematically.

Indicative results showed that participants perceived significant impacts in ability across eight key question groups evaluated. Differences were found between the perceived and observed impact in relation to technical areas covered within the programme which were perceived as the highest scoring impacts by participants.

The indicative results show a positive impact on practice has been both perceived and observed. Findings illustrate the value of this development programme on both the personal development of GP Mental Health Commissioners and commissioning practice. Although the findings of this evaluation increase understanding in relation to an important and topical area, larger scale, prospective evaluations are required. Impact evaluations could be embedded within future programmes to encourage higher participant and third-party engagement. Future evaluations would benefit from collection and analysis of attendance data. Further research could involve patient, service user and carer perspectives on mental health commissioning.

Results of this evaluation could inform the development of future learning programmes for mental health commissioners as part of a national approach to improve mental health provision.

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.

An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.

A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.

Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.

This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.

The purpose of this article is to analyze main principles, forms and approaches to education quality evaluation in the process of establishment, development and crucial changes in the state accreditation of educational institutions and study programmes in Russian higher education in the last 20 years.

The major research method used in the paper is the qualitative analysis of legal and statistical documents, research papers and accreditation practices, which impacted the development and transformation of accreditation forms in Russia.

The transformation process of state accreditation during the last 20 years was conditioned by the changes in the state education policy and socio-economic situation. In a short period, under the influence of internal and external factors, Russian higher education has experienced significant changes in the structure of higher education and quality assurance. This resulted in different approaches to accreditation: state and independent, mandatory and voluntary, national and international.

The research outcomes may be applicable in the countries with developing accreditation systems and comparable scope of education.

The study identifies the tendencies in the development of higher education and quality evaluation.

The paper systematizes the tendencies of development in quality assurance and distinguishes specific features and diversity of forms of the quality assurance in one of the largest systems of higher education.

This study aims to report the design, development and evaluation of a digital quality assurance application aimed at improving and ensuring the quality of assessment programmes in higher education.

The application was developed using a design-based research (DBR) methodology. The application’s design was informed by a literature search and needs assessment of quality assurance stakeholders to ensure compliance with daily practices and accreditation requirements. Stakeholders from three study programmes evaluated the application.

As part of the development of the application, module- and programme-level dashboards were created to provide an overview of the programme’s outcomes, assessment methods, assessment metrics, self-evaluated quality indicators and assessment documents. The application was evaluated by stakeholders at the module and programme levels. Overall, the results indicated that the dashboards aided them in gaining insight into the assessment programme and its alignment with underlying assessments.

Visualisation of the assessment programme’s structure and content identifies gaps and opportunities for improvement, which can be used to initiate a dialogue and further actions to improve assessment quality.

The application developed facilitates a cyclical and transparent assessment quality assurance procedure that is continuously available to various stakeholders in quality assurance.

One of the priorities of the health system is community health promotion. In this regards, proper development of programs and plans is needed to create a responsive system which leads to health promotion. The aim of this study was to identify the requirements for developing non-communicable disease (NCDs) programs based on CIPP (context, input, process and product) model.

This is a qualitative study. Data were collected through semi-structured interviews with 40 experienced informants in the field of NCDs. The interviewees were selected bases on pre-determined criteria which then were completed by snowball sampling. Analysis was carried out using a content analysis approach that led to identifying program development requirements in four dimensions of context, input, process and product.

Twenty-nine requirements of developing program were categorized in four domains of context, input, process and product. These requirements comprised of pilot studies, the existence of appropriate needs assessment, evidence-based programs, promoting organizational culture, adequacy of resources, identification of stakeholders and comprehensive cooperation and existence of an appropriate evaluation system.

Since this study was performed through a qualitative method, it is possible, some prerequisites of program development may not be encountered. But the extreme effort has been made to perceive diversity and different aspects.

The first study was in the field of appropriate requirements for program development in the context of a centralized health system in a developing country.

The purpose of this co-produced research project was to conduct interviews with people working in, volunteering with and accessing Canadian recovery colleges (RCs) to explore their perspectives on what an evaluation strategy for RCs could look like.

This study used a participatory action research approach and involved semistructured interviews with 29 people involved with RCs across Canada.

In this paper, the authors share insights from participants about the purposes of RC evaluation; key elements of evaluation; and the most applicable and effective approaches to evaluation. Participants indicated that RC evaluations should use a personalized, humanistic and accessible approach. The findings suggest that evaluations can serve multiple purposes and have the potential to support both organizational and personal-recovery goals if they are developed with meaningful input from people who access and work in RCs.

The findings can be used to guide evaluations in which aspects that are most important to those involved in RCs could inform choices, decisions, priorities, developments and adaptations in RC evaluation processes and, ultimately, in programming.

A recent scoping review revealed that although coproduction is a central feature of the RC model, coproduction principles are rarely acknowledged in descriptions of how RC evaluation strategies are developed. Exploring coproduction processes in all aspects of the RC model, including evaluation, can further the mission of RCs, which is to create spaces where people can come together and engage in mutual capacity-building and collaboration.

This paper aims to develop and present a new planning framework of social marketing, known as consumer research, segmentation, design of the social programme, implementation, evaluation and sustainability (CSD-IES).

The proposed framework is based on recent theoretical developments in social marketing and is informed by the key strengths of existing social marketing planning approaches.

The CSD-IES planning framework incorporates emerging principles of social marketing. For example, sustainability in changed behaviour, ethical considerations in designing social marketing programmes, the need for continuous research to understand the changing needs of the priority audience during the programme and the need for explicit feedback mechanisms.

The CSD-IES framework is a dynamic and flexible framework that guides social marketers, other practitioners and researchers to develop, implement and evaluate effective and sustainable social marketing programmes to influence or change specific behaviours based on available resources.

This paper makes an important contribution to social marketing theory and practice by integrating elements of behaviour maintenance, consideration of ethical perspectives and continuous feedback mechanisms in developing the CSD-IES framework, bringing it in line with the global consensus definition of social marketing.