Search results

1 – 10 of over 1000
Click here to view access options
Article
Publication date: 14 March 2016

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate…

Abstract

Purpose

There are five factors acting as a barrier to the effective evaluation of educational technology (edtech), which are as follows: premature timing, inappropriate techniques, rapid change, complexity of context and inconsistent terminology. The purpose of this paper is to identify new evaluation approaches that will address these and reflect on the evaluation imperative for complex technology initiatives.

Approach

An initial investigation of traditional evaluative approaches used within the technology domain was broadened to investigate the evaluation practices within social and public policy domains. Realist evaluation, a branch of theory-based evaluation, was identified and reviewed in detail. The realist approach was then refined, proposing two additional necessary steps to support mapping the technical complexity of initiatives.

Findings

A refined illustrative example of a realist evaluation framework is presented, including two novel architectural edtech domain reference models to support mapping.

Practical implications

Recommendations include building individual evaluator capacity; adopting the realist framework; the use of architectural edtech domain reference models; phased evaluation to first build theories in technology “context” and then iteratively during complex implementation chains; and community contribution to a shared map of technical and organisational complexity.

Originality

This paper makes a novel contribution by arguing the imperative for a theory-based realist approach to help redefine evaluative thinking within the IT and complex system domain. It becomes an innovative proposal with the addition of two domain reference models that tailor the approach for edtech. Its widespread adoption will help build a shared evidence base that synthesizes and surfaces “what works, for whom, in which contexts and why”, benefiting educators, IT managers, funders, policymakers and future learners.

Details

Journal of Systems and Information Technology, vol. 18 no. 1
Type: Research Article
ISSN: 1328-7265

Keywords

Click here to view access options
Article
Publication date: 16 June 2021

Alice Jones and Néstor Valero-Silva

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors…

Abstract

Purpose

English social housing providers are increasingly turning to social impact measurement to assess their social value. This paper aims to understand the contextual factors causing this rise in the practice, specifically within this sector; the mechanisms that enable it to be effectively implemented within an individual organisation and the outcomes of successful implementation for individual organisations and more widely across the sector and beyond.

Design/methodology/approach

A realist theory-based approach is applied to the study of a small number of social housing organisations and leaders within the sector to explore the use of social impact measurement. The paper addresses three questions: Why is social impact measurement being adopted in this sector? How is it successfully implemented? And what happens (outcomes) when it is successfully implemented? Addressing these questions necessitates deeper insight into the contextual pressures that have brought to the fore social impact measurement within the sector and the beneficial outcomes the practice provides (or is anticipated to provide) to social housing providers. The methodological approach of Realist Evaluation (Pawson and Tilley, 1997, 2004) is used to structure and analyse the empirical data and findings into a programme theory for social impact measurement. Realist Evaluation provides a programme theory perspective, seeking to answer the question “what works, for whom and in what circumstances?”. In this research, the “whom” refers to English social housing providers and the circumstances are the contextual conditions experienced by the sector over the past decade. The programme theory aims to set out the links between the contextual drivers for social impact measurement, the mechanisms that bring about its implementation and the outcomes that occur as a result. Within this, greater detail on the implementation perspective is provided by developing an implementation theory using a Theory of Change approach (Connell et al., 1995; Fulbright-Anderson et al., 1998). The implementation theory is then embedded within the wider programme theory so as to bring the two elements together, thereby creating a refinement of the overall theory for social impact measurement. In turn, this paper demonstrates its importance (the outcomes that it can achieve for organisations and the sector) and how it can effectively be implemented to bring about those outcomes.

Findings

Social housing providers use social impact measurement both internally, to determine their organisational priorities and externally, to demonstrate their value to local and national governments and cross-sector partners then to shape and influence resource allocation. The practice itself is shown to be an open and active programme, rather than a fixed calculative practice.

Research limitations/implications

The intensive nature of the research means that only a limited number of cases were explored. Further research could test theories developed here against evidence collected from a wider range of cases, e.g. other types of providers or non-adopters.

Practical implications

The research makes a strong contribution to practice in the form of a re-conceptualisation of how social impact measurement can be shown to be effective, based on a deeper understanding of causal mechanisms, how they interact and the outcomes that result. This is of value to the sector as such information could help other organisations both to understand the value of social impact measurement and to provide practical guidance on how to implement it effectively.

Social implications

As the practice of impact measurement continues to develop, practitioners will need to be aware of any changes to these contextual factors and consider questions such as: is the context still supportive of impact measurement? Does the practice need to be adjusted to meet the needs of the current context? For instance, the recent tragedy at Grenfell Tower has led to a reconsideration of the role of social housing; a new Green Paper is currently being drafted (Ministry for Housing, Communities and Local Government, 2018). This may have a number of implications for social impact measurement, such as a rebalancing of emphasis on outcomes relating to environmental improvements, towards outcomes relating to the well-being of tenants.

Originality/value

Existing literature is largely limited to technical guides. This paper links theory-based evaluation to practice contributing to social housing practice.

Details

Qualitative Research in Accounting & Management, vol. 18 no. 3
Type: Research Article
ISSN: 1176-6093

Keywords

Click here to view access options
Article
Publication date: 13 March 2017

Melanie Rose Nova King, Ray J. Dawson, Steve J. Rothberg and Firat Batmaz

This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations.

Downloads
256

Abstract

Purpose

This study aims to investigate the effectiveness of a theory-driven realist evaluative research approach to better understand complex technology implementations in organizations.

Design/methodology/approach

An institution wide e-learning implementation of lecture capture (LC), within a UK University, was chosen, and a realist evaluation framework was used, tailored for educational technology. The research was conducted over four, increasingly focused, evaluation cycles combining engagement analytics, user interviews and theory to refine what works (or does not work), for whom, in which contexts and why.

Findings

Despite explicit demand and corresponding investment, overall student engagement is lower than expected. Increased student use appears linked to particular staff attitudes and behaviours and not to specific disciplines or course content. The main benefits of LC are providing reassurance to the majority, aiding revision and understanding for the many and enabling catch-up for the few. Recommendations for future research are based on some unexpected outcomes uncovered, including evolving detrimental student behaviours, policy development based on technological determinism and future learner-centred system development for next-generation LC technologies.

Practical implications

The realist approach taken, and evaluation framework used, can be adopted (and adapted) for future evaluative research. Domain specific reference models, categorizing people and technology, supported analysis across multiple contexts.

Originality/value

This study responds to a call for more theory-based research in the field of educational technology. The authors demonstrate that a theory-driven approach provides real and practical recommendations for institutions and allows for greater insight into the political, economic and social complexity of technology implementation.

Details

Journal of Systems and Information Technology, vol. 19 no. 1/2
Type: Research Article
ISSN: 1328-7265

Keywords

Click here to view access options
Article
Publication date: 30 April 2019

Tom Grimwood

The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care…

Abstract

Purpose

The purpose of this paper is to discuss the methodological challenges to evaluating one of the 50 vanguard sites of the new care model (NCM) programme for integrated care in England, and make the case for a modified realist approach to this kind of evaluation.

Design/methodology/approach

The paper considers three challenges to evaluating the NCM in this particular vanguard: complexity, strategy and rhetoric. It reflects on how the realist approach negotiates these philosophical challenges to delivering integrated care, in order to provide contextualised accounts of who a programme works for, in what context, and why.

Findings

The paper argues that, in the case of this particular vanguard site, the tangible benefit of the realist approach was not in providing a firm epistemological basis for evaluation, but rather in drawing out and articulating the ontological rhetoric of such large-scale transformation programmes. By understanding the work of the NCM less as an objective “system”, and more as a dynamic form of persuasion, aimed at securing the “adherence of minds” (Perelman and Olbrechts-Tyteca, 2008, p. 8) in multiple audiences, the paper suggests that realist evaluation can be used to address both the systematic issues and localised successes the NCMs encountered.

Originality/value

The paper identifies a number of aspects of new models of integrated care for evaluators to consider. It offers ways of negotiating the challenges to conventional outcome-focused evaluation, by drawing attention to the need for contextualised, time-situated and audience-sensitive value of NCMs.

Details

Journal of Integrated Care, vol. 27 no. 3
Type: Research Article
ISSN: 1476-9018

Keywords

Click here to view access options
Article
Publication date: 2 May 2017

Avril Blamey, Jacki Gordon, Kim Newstead and Jacqueline McDowell

The purpose of this paper is to present learning on the strategies used by cooking skills practitioners and the programme theories, behaviour change mechanisms/contexts…

Abstract

Purpose

The purpose of this paper is to present learning on the strategies used by cooking skills practitioners and the programme theories, behaviour change mechanisms/contexts and intended outcomes associated with these in varied contexts.

Design/methodology/approach

Grey literature from Scottish cooking skills courses were reviewed using realist principles. Intervention implementation variables were identified and iteratively coded to uncover intended intervention strategies and programme theories. The lack of robust evaluation processes and outcome data in the grey literature prevented the testing of intended programme theories against outcomes. Alternatively, implementation strategies were aligned against behavioural-theory constructs contained in national guidance. Prioritised theories were further clarified/refined using practitioner and participant focus group data. Learning was used to inform future practice/evaluation.

Findings

Courses targeted and reached vulnerable individuals. Practitioners articulated multiple theories and assumptions about how strategies may work. Numerous strategies and behaviour constructs were used to target, tailor and reinforce cooking/food and wider social outcomes. Mechanisms were assumed to be triggered by different contexts and lead to varied outcomes. Strategies used were consistent with evidenced behaviour change constructs and guidelines. Interventions aimed to achieve non-cooking/social outcomes as well as cooking ones – including potential mediators of cooking behaviour, e.g. self-confidence. Contexts facilitated/limited the use of certain strategies. Limitations in course design, reporting and self-evaluation need to be addressed.

Practical implications

Recommendations for improving intervention commissioning, design and evaluation using realist principles are provided.

Originality/value

Learning addresses gaps in knowledge about the implementation of cooking skills interventions identified from systematic reviews and can improve course design and evaluation.

Details

British Food Journal, vol. 119 no. 5
Type: Research Article
ISSN: 0007-070X

Keywords

Click here to view access options
Article
Publication date: 3 February 2021

Francesca Caló, Michael James Roy, Cam Donaldson, Simon Teasdale and Simone Baglioni

As the provision of public services in many advanced welfare states has increasingly come to be marked by competition, social enterprises have actively been encouraged by…

Abstract

Purpose

As the provision of public services in many advanced welfare states has increasingly come to be marked by competition, social enterprises have actively been encouraged by governments to become involved in the delivery of public services. While the evaluation of complex public health interventions has arguably become increasingly more sophisticated, this has not been the case where social enterprise is concerned: evaluation of the actual impacts of social enterprises remains significantly underdeveloped by comparison. This study aims to support the establishment of a robust evidence base for the use of social enterprise as a policy instrument.

Design/methodology/approach

This paper assesses the potential of three methodological approaches common in the evaluation of complex public health interventions and applies them to the complex realm of community-led social enterprise.

Findings

Only through the involvement of different comparator groups, based on the research questions addressed, would it be possible to disentangle the embedded characteristics of organisations such as social enterprises. Each of the methods adopted in this research is time-consuming and resource-intensive and requires the researcher to possess advanced skills. Public officials should recognise the complexity and resource-intensive nature of such evaluation and resource it accordingly. If the aim of policymakers is to understand the added value of social enterprise organisations, an integrative research approach combining different research methods and design should be implemented to improve generalisability.

Originality/value

This study applies a range of favoured approaches to evaluate “complex” public health interventions include systematic reviews, realist evaluation and quasi-experimental investigation. However, such evaluation approaches have rarely been applied before in the context of social enterprise.

Details

Social Enterprise Journal, vol. 17 no. 1
Type: Research Article
ISSN: 1750-8614

Keywords

Click here to view access options
Book part
Publication date: 10 December 2018

Catherine Brentnall, Iván Diego Rodríguez and Nigel Culkin

The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and…

Abstract

The demand for including enterprise in the education system, at all levels and for all pupils is now a global phenomenon. Within this context, the use of competitions and competitive learning activities is presented as a popular and effective vehicle for learning. The purpose of this chapter is to illustrate how a realist method of enquiry – which utilises theory as the unit of analysis – can shed new light on the assumed and unintended outcomes of enterprise education competitions. The case developed here is that there are inherent flaws in assuming that competitions will ‘work’ in the ways set out in policy and guidance. Some of the most prevalent stated outcomes – that competitions will motivate and reward young people, that they will enable the development of entrepreneurial skills, and that learners will be inspired by their peers – are challenged by theory from psychology and education. The issue at stake is that the expansion of enterprise education policy into primary and secondary education increases the likelihood that more learners will be sheep dipped in competitions, and competitive activities, without a clear recognition of the potential unintended effects. In this chapter, we employ a realist-informed approach to critically evaluate the theoretical basis that underpins the use of competitions and competitive learning activities in school-based enterprise education. We believe that our findings and subsequent recommendations will provide those who promote and practice the use of competitions with a richer, more sophisticated picture of the potential flaws within such activities.

Details

Creating Entrepreneurial Space: Talking Through Multi-Voices, Reflections on Emerging Debates
Type: Book
ISBN: 978-1-78756-372-8

Keywords

Click here to view access options
Article
Publication date: 21 March 2016

Donald Forrester

Downloads
116

Abstract

Details

Journal of Children's Services, vol. 11 no. 1
Type: Research Article
ISSN: 1746-6660

Keywords

Abstract

Purpose

To ensure that more people will benefit from integrated care initiatives, scaling-up of successful initiatives is the way forward. However, new challenges present themselves as knowledge on how to achieve successful large-scale implementation is scarce. The EU-funded project SCIROCCO uses a step-based scaling-up strategy to explore what to scale-up, and how to scale-up integrated care initiatives by matching the complementary strengths and weaknesses of five European regions involved in integrated care. The purpose of this paper is to describe a multi-method evaluation protocol designed to understand what factors influence the implementation of the SCIROCCO strategy to support the scaling-up of integrated care.

Design/methodology/approach

The first part of the protocol focuses on the assessment of the implementation fidelity of the SCIROCCO step-based strategy. The objective is to gain insight in whether the step-based strategy is implemented as it was designed to explore what works and does not work when implementing the scaling-up strategy. The second part concerns a realist evaluation to examine what it is about the SCIROCCO’s strategy that works for whom, why, how and in which circumstances when scaling-up integrated care.

Findings

The intended study will provide valuable information on the implementation of the scaling-up strategy which will help to explain for what specific reasons the implementation succeeds and will facilitate further improvement of project outcomes.

Originality/value

The expected insights could be useful to guide the development, implementation and evaluation of future scaling-up strategies to advance the change towards more sustainable health and care systems.

Details

Journal of Integrated Care, vol. 27 no. 3
Type: Research Article
ISSN: 1476-9018

Keywords

Click here to view access options
Article
Publication date: 7 January 2019

Ryan Armstrong

The strategy map represents a major contribution to the theory and practice of performance management. However, it has failed to realize its full potential due to a lack…

Downloads
1162

Abstract

Purpose

The strategy map represents a major contribution to the theory and practice of performance management. However, it has failed to realize its full potential due to a lack of theoretical and conceptual development. Therefore, the purpose of this paper is to revisit the theories of strategy maps to better understand how and in what circumstances they benefit performance management.

Design/methodology/approach

The study employs realist synthesis, a method of systematic literature review. A theory on how strategy maps work is extracted from performance management literature, which are subsequently evaluated through a critical examination of empirical studies.

Findings

A theory of how strategy maps are meant work is presented in relation to the generic performance management stages of problem structuring, development and use, where they can serve as a tool for discovery and by stimulating social interactions. Based on the findings, 12 propositions are offered related to the effective use of strategy maps within a performance management framework.

Research limitations/implications

The introduction of the strategy map to performance management represented a breakthrough in how organizational performance could be understood and communicated. This study goes a step further by considering how they work and in what circumstances. In so doing, the study aims to open the way for new and more effective applications of strategy maps within the changing performance management context.

Practical implications

This study provides practitioners with actionable propositions which can help in effectively using strategy maps.

Originality/value

Distinguishing the aims and mechanisms of the strategy map along performance management systems has the potential to greatly increase their effectiveness in practice as a powerful, but underutilized tool. This paper also demonstrates how realist synthesis, currently an uncommon method in management studies, facilitated the creation of a new perspective of strategy maps to fit specifically within performance management.

Details

International Journal of Productivity and Performance Management, vol. 68 no. 4
Type: Research Article
ISSN: 1741-0401

Keywords

1 – 10 of over 1000