Search results
1 – 10 of over 2000
There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid…
Abstract
Purpose
There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.
Approach
An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.
Findings
A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.
Practical implications
Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.
Originality
This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.
Details
Keywords
Alice Jones and Néstor Valero-Silva
English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing…
Abstract
Purpose
English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.
Design/methodology/approach
A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.
Findings
Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.
Research limitations/implications
The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.
Practical implications
The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.
Social implications
As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.
Originality/value
Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.
Details
Keywords
Melanie Rose Nova King, Ray J. Dawson, Steve J. Rothberg and Firat Batmaz
This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations.
Abstract
Purpose
This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations.
Design/methodology/approach
An institution wide e-learning implementation of lecture capture (LC), within a UK University, was chosen, and a realist evaluation framework was used, tailored for educational technology. The research was conducted over four, increasingly focused, evaluation cycles combining engagement analytics, user interviews and theory to refine what works (or does not work), for whom, in which contexts and why.
Findings
Despite explicit demand and corresponding investment, overall student engagement is lower than expected. Increased student use appears linked to particular staff attitudes and behaviours and not to specific disciplines or course content. The main benefits of LC are providing reassurance to the majority, aiding revision and understanding for the many and enabling catch-up for the few. Recommendations for future research are based on some unexpected outcomes uncovered, including evolving detrimental student behaviours, policy development based on technological determinism and future learner-centred system development for next-generation LC technologies.
Practical implications
The realist approach taken, and evaluation framework used, can be adopted (and adapted) for future evaluative research. Domain specific reference models, categorizing people and technology, supported analysis across multiple contexts.
Originality/value
This study responds to a call for more theory-based research in the field of educational technology. The authors demonstrate that a theory-driven approach provides real and practical recommendations for institutions and allows for greater insight into the political, economic and social complexity of technology implementation.
Details
Keywords
Robert Wapshott and Oliver Mallett
This chapter argues for the unrealised potential value of methodologies derived from a critical realist research philosophy in the field of entrepreneurship studies. Critical…
Abstract
This chapter argues for the unrealised potential value of methodologies derived from a critical realist research philosophy in the field of entrepreneurship studies. Critical realism offers methodological alternatives that, through the generation of new insights into social relations, social structures and key generative mechanisms, can offer significant value for entrepreneurship researchers. Reflecting on their personal experiences researching from a critical realist perspective in entrepreneurship studies, the authors explore how this research philosophy can extend the field of inquiry and promote new perspectives. The chapter explores this in relation to the specific topic of enterprise policy and demystifies some aspects of critical realism by setting out some of its basic principles to demonstrate their potential to develop new insights. Further, this approach can create significant impact, for example, through the development of effective interventions. The chapter concludes by identifying implications for enterprise policy development, implementation and evaluation.
Details
Keywords
Stephen Case, Charlie E. Sutton, Joanne Greenhalgh, Mark Monaghan and Judy Wright
This study aims to examine the extent to which “What Works” reviews in youth justice enable understanding of the features of effectiveness (what works, for whom, in what…
Abstract
Purpose
This study aims to examine the extent to which “What Works” reviews in youth justice enable understanding of the features of effectiveness (what works, for whom, in what circumstances and why?) specified in the Effects–Mechanisms–Moderators–Implementation–Economic cost (EMMIE) framework.
Design/methodology/approach
The EMMIE framework examined findings within a sample of “What Works” style reviews of preventative youth justice intervention effectiveness.
Findings
“What Works” style reviews of evaluations of preventative youth justice interventions often omit the requisite details required to examine all of the necessary elements of effectiveness contained within the EMMIE framework. While effectiveness measures were typically provided, the dominant evaluation evidence-base struggles to consider moderators of effect, mechanisms of change, implementation differences and cost-effectiveness. Therefore, “What Works” samples cannot facilitate sufficient understanding of “what works for whom, in what circumstances and why?”. The authors argue that Realist Synthesis can fill this gap and shed light on the contexts that shape the mechanisms through which youth justice interventions work.
Originality/value
The authors extended the approach adopted by an earlier review of effectiveness reviews (Tompson et al., 2020), considering more recent reviews of the effectiveness of preventative interventions using the EMMIE framework. Unlike previous reviews, the authors prioritised the utility of the EMMIE framework for assessing the factors affecting the effectiveness of preventative interventions in youth justice.
Details
Keywords
The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care in…
Abstract
Purpose
The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care in England, and make the case for a modified realist approach to this kind of evaluation.
Design/methodology/approach
The paper considers three challenges to evaluating the NCM in this particular vanguard: complexity, strategy and rhetoric. It reflects on how the realist approach negotiates these philosophical challenges to delivering integrated care, in order to provide contextualised accounts of who a programme works for, in what context, and why.
Findings
The paper argues that, in the case of this particular vanguard site, the tangible benefit of the realist approach was not in providing a firm epistemological basis for evaluation, but rather in drawing out and articulating the ontological rhetoric of such large-scale transformation programmes. By understanding the work of the NCM less as an objective “system”, and more as a dynamic form of persuasion, aimed at securing the “adherence of minds” (Perelman and Olbrechts-Tyteca, 2008, p. 8) in multiple audiences, the paper suggests that realist evaluation can be used to address both the systematic issues and localised successes the NCMs encountered.
Originality/value
The paper identifies a number of aspects of new models of integrated care for evaluators to consider. It offers ways of negotiating the challenges to conventional outcome-focused evaluation, by drawing attention to the need for contextualised, time-situated and audience-sensitive value of NCMs.
Details
Keywords
Avril Blamey, Jacki Gordon, Kim Newstead and Jacqueline McDowell
The purpose of this paper is to present learning on the strategies used by cooking skills practitioners and the programme theories, behaviour change mechanisms/contexts and…
Abstract
Purpose
The purpose of this paper is to present learning on the strategies used by cooking skills practitioners and the programme theories, behaviour change mechanisms/contexts and intended outcomes associated with these in varied contexts.
Design/methodology/approach
Grey literature from Scottish cooking skills courses were reviewed using realist principles. Intervention implementation variables were identified and iteratively coded to uncover intended intervention strategies and programme theories. The lack of robust evaluation processes and outcome data in the grey literature prevented the testing of intended programme theories against outcomes. Alternatively, implementation strategies were aligned against behavioural-theory constructs contained in national guidance. Prioritised theories were further clarified/refined using practitioner and participant focus group data. Learning was used to inform future practice/evaluation.
Findings
Courses targeted and reached vulnerable individuals. Practitioners articulated multiple theories and assumptions about how strategies may work. Numerous strategies and behaviour constructs were used to target, tailor and reinforce cooking/food and wider social outcomes. Mechanisms were assumed to be triggered by different contexts and lead to varied outcomes. Strategies used were consistent with evidenced behaviour change constructs and guidelines. Interventions aimed to achieve non-cooking/social outcomes as well as cooking ones – including potential mediators of cooking behaviour, e.g. self-confidence. Contexts facilitated/limited the use of certain strategies. Limitations in course design, reporting and self-evaluation need to be addressed.
Practical implications
Recommendations for improving intervention commissioning, design and evaluation using realist principles are provided.
Originality/value
Learning addresses gaps in knowledge about the implementation of cooking skills interventions identified from systematic reviews and can improve course design and evaluation.
Details
Keywords
Francesca Caló, Michael James Roy, Cam Donaldson, Simon Teasdale and Simone Baglioni
As the provision of public services in many advanced welfare states has increasingly come to be marked by competition, social enterprises have actively been encouraged by…
Abstract
Purpose
As the provision of public services in many advanced welfare states has increasingly come to be marked by competition, social enterprises have actively been encouraged by governments to become involved in the delivery of public services. While the evaluation of complex public health interventions has arguably become increasingly more sophisticated, this has not been the case where social enterprise is concerned: evaluation of the actual impacts of social enterprises remains significantly underdeveloped by comparison. This study aims to support the establishment of a robust evidence base for the use of social enterprise as a policy instrument.
Design/methodology/approach
This paper assesses the potential of three methodological approaches common in the evaluation of complex public health interventions and applies them to the complex realm of community-led social enterprise.
Findings
Only through the involvement of different comparator groups, based on the research questions addressed, would it be possible to disentangle the embedded characteristics of organisations such as social enterprises. Each of the methods adopted in this research is time-consuming and resource-intensive and requires the researcher to possess advanced skills. Public officials should recognise the complexity and resource-intensive nature of such evaluation and resource it accordingly. If the aim of policymakers is to understand the added value of social enterprise organisations, an integrative research approach combining different research methods and design should be implemented to improve generalisability.
Originality/value
This study applies a range of favoured approaches to evaluate “complex” public health interventions include systematic reviews, realist evaluation and quasi-experimental investigation. However, such evaluation approaches have rarely been applied before in the context of social enterprise.
Details
Keywords
Hamid Roodbari, Karina Nielsen, Carolyn Axtell, Susan E. Peters and Glorian Sorensen
Realist evaluation seeks to answer the question of “what works for whom in which circumstances?” through developing and testing middle range theories (MRTs). MRTs are programme…
Abstract
Purpose
Realist evaluation seeks to answer the question of “what works for whom in which circumstances?” through developing and testing middle range theories (MRTs). MRTs are programme theories that outline how certain mechanisms of an intervention work in a specific context to bring about certain outcomes. In this paper, the authors tested an initial MRT about the mechanism of participation. The authors used evidence from a participatory organisational intervention in five worksites of a large multi-national organisation in the US food service industry.
Design/methodology/approach
Qualitative data from 89 process tracking documents and 24 post-intervention, semi-structured interviews with intervention stakeholders were analysed using template analysis.
Findings
The operationalised mechanism was partial worksite managers’ engagement with the research team. Six contextual factors (e.g. high workload) impaired participation, and one contextual factor (i.e. existing participatory practices) facilitated participation. Worksite managers’ participation resulted in limited improvement in their awareness of how working conditions can impact on their employees’ safety, health, and well-being. Based on these findings, the authors modified the initial MRT into an empirical MRT.
Originality/value
This paper contributes to the understanding of “what works for whom in which circumstances” regarding participation in organisational interventions.
Details
Keywords
Catherine Brentnall, Iván Diego Rodríguez and Nigel Culkin
The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and…
Abstract
The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and competitive learning activities is presented as a popular and effective vehicle for learning. The purpose of this chapter is to illustrate how a realist method of enquiry – which utilises theory as the unit of analysis – can shed new light on the assumed and unintended outcomes of enterprise education competitions. The case developed here is that there are inherent flaws in assuming that competitions will ‘work’ in the ways set out in policy and guidance. Some of the most prevalent stated outcomes – that competitions will motivate and reward young people, that they will enable the development of entrepreneurial skills, and that learners will be inspired by their peers – are challenged by theory from psychology and education. The issue at stake is that the expansion of enterprise education policy into primary and secondary education increases the likelihood that more learners will be sheep dipped in competitions, and competitive activities, without a clear recognition of the potential unintended effects. In this chapter, we employ a realist-informed approach to critically evaluate the theoretical basis that underpins the use of competitions and competitive learning activities in school-based enterprise education. We believe that our findings and subsequent recommendations will provide those who promote and practice the use of competitions with a richer, more sophisticated picture of the potential flaws within such activities.
Details