Search results

1 – 10 of over 2000
Content available
Article
Publication date: 14 March 2016

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid…

Abstract

Purpose

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.

Approach

An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.

Findings

A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.

Practical implications

Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.

Originality

This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.

Details

Journal of Systems and Information Technology, vol. 18 no. 1
Type: Research Article
ISSN: 1328-7265

Keywords

Article
Publication date: 16 June 2021

Alice Jones and Néstor Valero-Silva

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing…

Abstract

Purpose

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.

Design/methodology/approach

A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.

Findings

Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.

Research limitations/implications

The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.

Practical implications

The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.

Social implications

As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.

Originality/value

Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.

Details

Qualitative Research in Accounting & Management, vol. 18 no. 3
Type: Research Article
ISSN: 1176-6093

Keywords

Article
Publication date: 13 March 2017

Melanie Rose Nova King, Ray J. Dawson, Steve J. Rothberg and Firat Batmaz

This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations.

317

Abstract

Purpose

This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations.

Design/methodology/approach

An institution wide e-learning implementation of lecture capture (LC), within a UK University, was chosen, and a realist evaluation framework was used, tailored for educational technology. The research was conducted over four, increasingly focused, evaluation cycles combining engagement analytics, user interviews and theory to refine what works (or does not work), for whom, in which contexts and why.

Findings

Despite explicit demand and corresponding investment, overall student engagement is lower than expected. Increased student use appears linked to particular staff attitudes and behaviours and not to specific disciplines or course content. The main benefits of LC are providing reassurance to the majority, aiding revision and understanding for the many and enabling catch-up for the few. Recommendations for future research are based on some unexpected outcomes uncovered, including evolving detrimental student behaviours, policy development based on technological determinism and future learner-centred system development for next-generation LC technologies.

Practical implications

The realist approach taken, and evaluation framework used, can be adopted (and adapted) for future evaluative research. Domain specific reference models, categorizing people and technology, supported analysis across multiple contexts.

Originality/value

This study responds to a call for more theory-based research in the field of educational technology. The authors demonstrate that a theory-driven approach provides real and practical recommendations for institutions and allows for greater insight into the political, economic and social complexity of technology implementation.

Details

Journal of Systems and Information Technology, vol. 19 no. 1/2
Type: Research Article
ISSN: 1328-7265

Keywords

Book part
Publication date: 10 November 2023

Robert Wapshott and Oliver Mallett

This chapter argues for the unrealised potential value of methodologies derived from a critical realist research philosophy in the field of entrepreneurship studies. Critical…

Abstract

This chapter argues for the unrealised potential value of methodologies derived from a critical realist research philosophy in the field of entrepreneurship studies. Critical realism offers methodological alternatives that, through the generation of new insights into social relations, social structures and key generative mechanisms, can offer significant value for entrepreneurship researchers. Reflecting on their personal experiences researching from a critical realist perspective in entrepreneurship studies, the authors explore how this research philosophy can extend the field of inquiry and promote new perspectives. The chapter explores this in relation to the specific topic of enterprise policy and demystifies some aspects of critical realism by setting out some of its basic principles to demonstrate their potential to develop new insights. Further, this approach can create significant impact, for example, through the development of effective interventions. The chapter concludes by identifying implications for enterprise policy development, implementation and evaluation.

Details

Nurturing Modalities of Inquiry in Entrepreneurship Research: Seeing the World Through the Eyes of Those Who Research
Type: Book
ISBN: 978-1-80262-186-0

Keywords

Article
Publication date: 9 September 2022

Stephen Case, Charlie E. Sutton, Joanne Greenhalgh, Mark Monaghan and Judy Wright

This study aims to examine the extent to which “What Works” reviews in youth justice enable understanding of the features of effectiveness (what works, for whom, in what…

Abstract

Purpose

This study aims to examine the extent to which “What Works” reviews in youth justice enable understanding of the features of effectiveness (what works, for whom, in what circumstances and why?) specified in the Effects–Mechanisms–Moderators–Implementation–Economic cost (EMMIE) framework.

Design/methodology/approach

The EMMIE framework examined findings within a sample of “What Works” style reviews of preventative youth justice intervention effectiveness.

Findings

“What Works” style reviews of evaluations of preventative youth justice interventions often omit the requisite details required to examine all of the necessary elements of effectiveness contained within the EMMIE framework. While effectiveness measures were typically provided, the dominant evaluation evidence-base struggles to consider moderators of effect, mechanisms of change, implementation differences and cost-effectiveness. Therefore, “What Works” samples cannot facilitate sufficient understanding of “what works for whom, in what circumstances and why?”. The authors argue that Realist Synthesis can fill this gap and shed light on the contexts that shape the mechanisms through which youth justice interventions work.

Originality/value

The authors extended the approach adopted by an earlier review of effectiveness reviews (Tompson et al., 2020), considering more recent reviews of the effectiveness of preventative interventions using the EMMIE framework. Unlike previous reviews, the authors prioritised the utility of the EMMIE framework for assessing the factors affecting the effectiveness of preventative interventions in youth justice.

Details

Safer Communities, vol. 21 no. 4
Type: Research Article
ISSN: 1757-8043

Keywords

Article
Publication date: 30 April 2019

Tom Grimwood

The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care in…

Abstract

Purpose

The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care in England, and make the case for a modified realist approach to this kind of evaluation.

Design/methodology/approach

The paper considers three challenges to evaluating the NCM in this particular vanguard: complexity, strategy and rhetoric. It reflects on how the realist approach negotiates these philosophical challenges to delivering integrated care, in order to provide contextualised accounts of who a programme works for, in what context, and why.

Findings

The paper argues that, in the case of this particular vanguard site, the tangible benefit of the realist approach was not in providing a firm epistemological basis for evaluation, but rather in drawing out and articulating the ontological rhetoric of such large-scale transformation programmes. By understanding the work of the NCM less as an objective “system”, and more as a dynamic form of persuasion, aimed at securing the “adherence of minds” (Perelman and Olbrechts-Tyteca, 2008, p. 8) in multiple audiences, the paper suggests that realist evaluation can be used to address both the systematic issues and localised successes the NCMs encountered.

Originality/value

The paper identifies a number of aspects of new models of integrated care for evaluators to consider. It offers ways of negotiating the challenges to conventional outcome-focused evaluation, by drawing attention to the need for contextualised, time-situated and audience-sensitive value of NCMs.

Details

Journal of Integrated Care, vol. 27 no. 3
Type: Research Article
ISSN: 1476-9018

Keywords

Article
Publication date: 2 May 2017

Avril Blamey, Jacki Gordon, Kim Newstead and Jacqueline McDowell

The purpose of this paper is to present learning on the strategies used by cooking skills practitioners and the programme theories, behaviour change mechanisms/contexts and…

Abstract

Purpose

The purpose of this paper is to present learning on the strategies used by cooking skills practitioners and the programme theories, behaviour change mechanisms/contexts and intended outcomes associated with these in varied contexts.

Design/methodology/approach

Grey literature from Scottish cooking skills courses were reviewed using realist principles. Intervention implementation variables were identified and iteratively coded to uncover intended intervention strategies and programme theories. The lack of robust evaluation processes and outcome data in the grey literature prevented the testing of intended programme theories against outcomes. Alternatively, implementation strategies were aligned against behavioural-theory constructs contained in national guidance. Prioritised theories were further clarified/refined using practitioner and participant focus group data. Learning was used to inform future practice/evaluation.

Findings

Courses targeted and reached vulnerable individuals. Practitioners articulated multiple theories and assumptions about how strategies may work. Numerous strategies and behaviour constructs were used to target, tailor and reinforce cooking/food and wider social outcomes. Mechanisms were assumed to be triggered by different contexts and lead to varied outcomes. Strategies used were consistent with evidenced behaviour change constructs and guidelines. Interventions aimed to achieve non-cooking/social outcomes as well as cooking ones – including potential mediators of cooking behaviour, e.g. self-confidence. Contexts facilitated/limited the use of certain strategies. Limitations in course design, reporting and self-evaluation need to be addressed.

Practical implications

Recommendations for improving intervention commissioning, design and evaluation using realist principles are provided.

Originality/value

Learning addresses gaps in knowledge about the implementation of cooking skills interventions identified from systematic reviews and can improve course design and evaluation.

Details

British Food Journal, vol. 119 no. 5
Type: Research Article
ISSN: 0007-070X

Keywords

Article
Publication date: 19 July 2022

Hamid Roodbari, Karina Nielsen, Carolyn Axtell, Susan E. Peters and Glorian Sorensen

Realist evaluation seeks to answer the question of “what works for whom in which circumstances?” through developing and testing middle range theories (MRTs). MRTs are programme…

Abstract

Purpose

Realist evaluation seeks to answer the question of “what works for whom in which circumstances?” through developing and testing middle range theories (MRTs). MRTs are programme theories that outline how certain mechanisms of an intervention work in a specific context to bring about certain outcomes. In this paper, the authors tested an initial MRT about the mechanism of participation. The authors used evidence from a participatory organisational intervention in five worksites of a large multi-national organisation in the US food service industry.

Design/methodology/approach

Qualitative data from 89 process tracking documents and 24 post-intervention, semi-structured interviews with intervention stakeholders were analysed using template analysis.

Findings

The operationalised mechanism was partial worksite managers’ engagement with the research team. Six contextual factors (e.g. high workload) impaired participation, and one contextual factor (i.e. existing participatory practices) facilitated participation. Worksite managers’ participation resulted in limited improvement in their awareness of how working conditions can impact on their employees’ safety, health, and well-being. Based on these findings, the authors modified the initial MRT into an empirical MRT.

Originality/value

This paper contributes to the understanding of “what works for whom in which circumstances” regarding participation in organisational interventions.

Details

International Journal of Workplace Health Management, vol. 15 no. 6
Type: Research Article
ISSN: 1753-8351

Keywords

Article
Publication date: 3 February 2021

Francesca Caló, Michael James Roy, Cam Donaldson, Simon Teasdale and Simone Baglioni

As the provision of public services in many advanced welfare states has increasingly come to be marked by competition, social enterprises have actively been encouraged by…

Abstract

Purpose

As the provision of public services in many advanced welfare states has increasingly come to be marked by competition, social enterprises have actively been encouraged by governments to become involved in the delivery of public services. While the evaluation of complex public health interventions has arguably become increasingly more sophisticated, this has not been the case where social enterprise is concerned: evaluation of the actual impacts of social enterprises remains significantly underdeveloped by comparison. This study aims to support the establishment of a robust evidence base for the use of social enterprise as a policy instrument.

Design/methodology/approach

This paper assesses the potential of three methodological approaches common in the evaluation of complex public health interventions and applies them to the complex realm of community-led social enterprise.

Findings

Only through the involvement of different comparator groups, based on the research questions addressed, would it be possible to disentangle the embedded characteristics of organisations such as social enterprises. Each of the methods adopted in this research is time-consuming and resource-intensive and requires the researcher to possess advanced skills. Public officials should recognise the complexity and resource-intensive nature of such evaluation and resource it accordingly. If the aim of policymakers is to understand the added value of social enterprise organisations, an integrative research approach combining different research methods and design should be implemented to improve generalisability.

Originality/value

This study applies a range of favoured approaches to evaluate “complex” public health interventions include systematic reviews, realist evaluation and quasi-experimental investigation. However, such evaluation approaches have rarely been applied before in the context of social enterprise.

Details

Social Enterprise Journal, vol. 17 no. 1
Type: Research Article
ISSN: 1750-8614

Keywords

Book part
Publication date: 10 December 2018

Catherine Brentnall, Iván Diego Rodríguez and Nigel Culkin

The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and…

Abstract

The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and competitive learning activities is presented as a popular and effective vehicle for learning. The purpose of this chapter is to illustrate how a realist method of enquiry – which utilises theory as the unit of analysis – can shed new light on the assumed and unintended outcomes of enterprise education competitions. The case developed here is that there are inherent flaws in assuming that competitions will ‘work’ in the ways set out in policy and guidance. Some of the most prevalent stated outcomes – that competitions will motivate and reward young people, that they will enable the development of entrepreneurial skills, and that learners will be inspired by their peers – are challenged by theory from psychology and education. The issue at stake is that the expansion of enterprise education policy into primary and secondary education increases the likelihood that more learners will be sheep dipped in competitions, and competitive activities, without a clear recognition of the potential unintended effects. In this chapter, we employ a realist-informed approach to critically evaluate the theoretical basis that underpins the use of competitions and competitive learning activities in school-based enterprise education. We believe that our findings and subsequent recommendations will provide those who promote and practice the use of competitions with a richer, more sophisticated picture of the potential flaws within such activities.

Details

Creating Entrepreneurial Space: Talking Through Multi-Voices, Reflections on Emerging Debates
Type: Book
ISBN: 978-1-78756-372-8

Keywords

1 – 10 of over 2000