Search results

A meta-evaluation is an assessment of evaluation practices. Meta-evaluations include assessments of validity and usefulness of two or more studies that focus on the same issues. Every performance audit is grounded explicitly or implicitly in one or more theories of program evaluation. A deep understanding of alternative theories of program evaluation is helpful to gain clarity about sound auditing practices. We present a review of several theories of program evaluation.

This study includes a meta-evaluation of seven government audits on the efficiency and effectiveness of tourism departments and programs. The seven tourism-marketing performance audits are program evaluations for: Missouri, North Carolina, Tennessee, Minnesota, Australia, and two for Hawaii. The majority of these audits are negative performance assessments. Similarly, although these audits are more useful than none at all, the central conclusion of the meta-evaluation is that most of these audit reports are inadequate assessments. These audits are too limited in the issues examined; not sufficiently grounded in relevant evaluation theory and practice; and fail to include recommendations, that if implemented, would result in substantial increases in performance.

This paper aims to describe a case between practitioners and social marketing academics to grow and scale a programme that engages with primary schools, teachers, children and the education network, inspiring students to become marine leaders and ocean champions.

Over a six-year period, the authors first applied collective intelligence to work with stakeholders across society to better understand the barriers and solutions to teaching children (6–12 year olds) about the ocean in schools. Following this, a Collective Impact Assessment of the Explorers Education Programme took place to grow the impact of the programme.

The Explorers Education Programme has grown its numbers higher than pre-pandemic levels. In 2022, the Explorers Education Programme had the largest number of participating children, reaching 15,237, with a growth of 21% compared to pre-pandemic levels in 2019 and 79% compared to 2021. In 2023, the programme won the “Best Education Outreach Award” category of the Education Awards in Ireland.

This research stresses the importance of measuring impact. The long-term impact of the Explorers Education Programme at societal, environmental and economical levels takes a much longer time frame to measure than the six years of these research collaborations.

The collaborative approach between academics and practitioners meant that this research had practical implications, whereby necessary and effective changes and learnings could be directly applied to the Explorers Education Programme in real time, as the practitioners involved were directly responsible for the management and coordination of the programme.

The value of collaborations and engagement between academia and practice cannot be underestimated. The ability to collectively reflect and assess impact moves beyond “an” intervention, allowing for more meaningful behavioural, social and system changes for the collective good, inspiring the next generation of marine leaders and ocean champions.

This chapter targets the learning of middle leaders working in the rapidly expanding international school sector in the Asia-Pacific Region. It draws on three externally commissioned impact studies of Leading Upstream (LU) – a purpose-designed 12-month part-time, leader learning program. The program runs in Hong Kong for middle leaders from 20 primary and secondary schools that make up a semigovernment education system. The main aim of the program was to scale up individual, team, and school capacity through a structured learning network design. Since 2005/2006, the program has completed four cohorts. The authors present an analysis of the impact of a connected series of the same program to draw insights that may inform program development for middle leaders. The heart of the chapter focuses on the authors’ attempts to synthesize the outcomes of the three impact studies. Data patterns from across the studies were analyzed to identify common patterns. Patterns determined were divided into personal, team/school, and system impact. Among the former is ‘increased confidence in self as leader” and the later the fragility of even moderately broad networks when learning hits the realities of school.

The purpose of this paper is to analyze the impact of microfinance programs on the income and food expenditure of farm and nonfarm households in Thailand.

The study employs secondary data from the Thai Socioeconomic Survey (cross-sectional data from 2017 and panel data from 2012 to 2017). The cross-sectional data (2017) include 43,210 households. Panel data from the 2012 and 2017 Socioeconomic surveys (SES surveys) include 4,406 households. The estimation methods include propensity score matching (PSM) and a fixed effect (FE) model.

The result shows that village funds (VFs) have a significant negative impact on income and food expenditure for both farm and nonfarm households. The empirical results reveal that the saving groups for production (SGPs) effects are positively significant in terms of income and food expenditure, but only for farm households. The FE model result also shows that while VFs have a negative impact on income they have a positive impact on food expenditure for farm households. In contrast, SPGs have no impact on both farm and nonfarm households' income and food expenditure.

Farm and nonfarm households require both welfare and microfinance programs. Microfinance programs can only help these households once they have the necessary education. The government should provide social programs and business skills for these households; completion of these courses should be a pre-requisite for accessing microfinance programs.

This study is unique because it reveals the microfinance impact between VFs and SGPs programs so that most low-income and poor people in Thailand can access basic financial services.

The purpose of this paper is to describe the evolving IFLA approach to impact evaluation through three of its international programmes: Freedom of Access to Information, Building Strong Library Associations (BSLA) and the International Advocacy Programme (IAP). This review positions these three programmes within the wider discourse of the international evaluation community.

Each of the three programmes is considered in turn to show what they were trying to achieve and how thinking about impact evaluation at IFLA is evolving.

This paper reports key evaluation findings for relevant phases of the BSLA and IAP programmes in general terms.

The views presented are those of the evaluation consultants who advised each of these programmes (and in the cases of BSLA and the IAP conducted the programme evaluations).

The processes described and the conclusions drawn should be of interest to anyone involved in international or national library evaluation, especially of public libraries, library associations and national libraries.

The paper suggests that more systematic impact evaluation of public libraries, library associations and national libraries is necessary to ensure their future survival.

The authors were uniquely placed to see and participate in IFLA impact evaluation discussions over the past decade.

This paper aims to present the impact evaluation findings from a multiprofessional leadership programme commissioned in the South East of England to support primary care networks (PCNs) to lead system improvement together. It identifies programme impact at micro and meso system levels; a leadership impact continuum that can be used by individuals and teams to evidence impact of improvements in PCN practices; the learning and development strategies that were effective and proposes implications for other networks.

Mixed methods underpinned by practice development methodology were used to explore the impact of the programme on two practitioner cohorts across 16 PCNs. Data were collected at the start, mid-point and end of the eight-month programme.

Results illustrate an innovative approach to collective leadership development. A continuum of impact created with participants offers insight into the journey of transformation, recognising that “change starts with me”. The impact framework identifies enablers, attributes and consequences for measuring and leading change at micro, meso and macro levels of the health-care system. Participants learned how to facilitate change and collaboratively solve problems through peer consulting which created a safe space for individuals to discuss workplace issues and receive multiprofessional views through action learning. These activities enabled teams to present innovative projects to commissioners for service redesign, enabling their PCN to be more effective in meeting population health needs. The authors believe that this programme may provide a model for other PCNs England and other place-based care systems internationally.

This study offers insight into how to enable a journey of transformation for individuals and PCN teams to enhance team effectiveness and collective leadership for system-wide transformation required by the National Health Service Long Term Plan (2019).

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.

A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.

Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.

The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.

The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.

As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.

Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.

There is conflicting evidence concerning the impact of hospital accreditation programmes, including across the Middle East Region, where such programmes have been most recently implemented in Iran, Jordan and Saudi Arabia. This paper maps available evidence regarding the impact of hospital accreditation in these three countries and draws attention to knowledge gaps for consideration.

This scoping review was conducted in 2020, using the Arksey and O'Malley framework. Five research databases were searched, along with five government and accreditation agency websites. Searches were complemented by citation chaining. English and Arabic publications evaluating hospital accreditation in the selected countries were included. Commentaries and articles not based on primary data collection and reviews of existing registry data were excluded. There were no exclusions based on study design or methods. A descriptive numerical summary and thematic analysis were used to synthesise the literature.

studies were included. The majority (n = 35) were published since 2014 and conducted in Saudi Arabia (n = 16). Four themes emerged: organisational impacts, patient safety, quality of care, and patient satisfaction and experience. The literature generally highlights positive impacts of accreditation, but most studies were based solely on health professionals' subjective perceptions. “Organisational impacts” had the largest, and strongest body of supporting evidence, while “patient safety” had the least and most variable evidence.

Opportunities to strengthen the design and evaluation of hospital accreditation programmes in the selected countries are highlighted. Additional experimental, mixed-method research is recommended to strengthen the evidence base and inform practical enhancements to hospital accreditation programmes in the region.

This study aims to illustrate the impact-assessment procedure of low-carbon tourism (LCT) program implemented in a world heritage city and to develop specific indicators toward sustainability.

The impact-assessment framework was indicator-based and designed for creating sustainable tourism (ST) in a case study. A set of indicators in various dimensions was developed and applied, referring to the UNWTO guideline. A mixed method of primary and secondary data collected from different sources included document review, site observation, key informant interview, questionnaire survey and focus-group discussions. Assessment of actual/observed impacts was proceeded based on the data collected from tourists and stakeholders, especially on tourist behaviors and resource consumptions.

The implementation of LCT program in a world heritage city provided impacts in different dimensions and characters. The observed activities were majorly tourism activities in accommodations and recreational places. The indicator initiation is the first development toward sustainability in a case of tourism study in a city destination. Indicators were developed with participation from key stakeholders and covered sustainability and carbon-emission dimensions. Impact-assessment results show a positive theme in less carbon emission, enhanced local income distribution and community capacity. However, the negative impacts include increased amounts of resource consumption and waste generation in visiting sites. The impact matrix works as the map for decision-makers to maximize benefits and manage the cons of the LCT program toward ST principles.

Research methodology, procedure and results on impact assessment with holistic perspectives imply academic contribution and practical benefits for decision-makers regarding ST development. The number of samples and enterprises was limited because of the program implementation period.

The research illustrates the impact-assessment process for an implemented city-based LCT program toward ST, where stakeholder participation was also functioning. A list of indicators was specially designed and can be practically applied for other LCT programs in city destinations. Applying a sustainability impact-assessment framework to the program can provide a clear presentation on how to develop ST.

This paper aims to examine how social marketing intervention programmes to measure, evaluate and document social marketing impact.

A systematic review of 49 nutritional behaviour intervention programmes (2006–2020) was conducted. To examine the social marketing impact of the programmes, a logic model of social impact was used. The model comprises inputs (the resources used for an intervention programme), outputs (the direct products resulting from the use of resources), outcomes (short- to medium-term programme effects) and impacts (long-term programme effects on the individual, community or societal levels).

Most intervention programmes set the goal of encouraging their target audience to increase fruit and vegetable intake, choose healthy food items, drink less sugary beverages or consume low-fat diaries, while few others sought policy or systems change. Multiple criteria were used for impact evaluation (e.g. exposure and reach, changes in knowledge, awareness, attitudes, behaviours and body mass index). (Quasi) experiments were the most popular method used for impact measurement, followed by the pre-post model of impact. Positive changes were found in 33 programmes, often reported in terms of short-term outputs or outcomes. Long-term impact particularly on the broader societal level was not indicated.

This research offers a systematic review of how social marketing impact is measured, evaluated and documented. It also provides some guidance for social marketers on how to shift from a reductionist, behavioural outcome-focussed approach towards an “expansionist” impact approach that explicitly considers social marketing impacts on the quality of life of individuals, communities and societies.