Citation

Ventista, O.M. (2021), "Prelims", How to Evaluate the Effectiveness of a School-Based Intervention: Evaluating the Impact of the Philosophy for Children Programme on Students' Skills, Emerald Publishing Limited, Leeds, pp. i-xviii. https://doi.org/10.1108/978-1-80043-002-020211013

Publisher

:

Emerald Publishing Limited

Copyright © 2021 Ourania Maria Ventista


Half Title Page

How to Evaluate the Effectiveness of a School-Based Intervention

Title Page

How to Evaluate the Effectiveness of a School-Based Intervention: Evaluating the Impact of the Philosophy for Children Programme on Students’ Skills

BY

OURANIA MARIA VENTISTA

United Kingdom – North America – Japan – India – Malaysia – China

Copyright Page

Emerald Publishing Limited

Howard House, Wagon Lane, Bingley BD16 1WA, UK

First edition 2021

Copyright © 2021 Ourania Maria Ventista. Published under exclusive license by Emerald Publishing Limited.

Reprints and permissions service

Contact:

No part of this book may be reproduced, stored in a retrieval system, transmitted in any form or by any means electronic, mechanical, photocopying, recording or otherwise without either the prior written permission of the publisher or a licence permitting restricted copying issued in the UK by The Copyright Licensing Agency and in the USA by The Copyright Clearance Center. Any opinions expressed in the chapters are those of the authors. Whilst Emerald makes every effort to ensure the quality and accuracy of its content, Emerald makes no representation implied or otherwise, as to the chapters’ suitability and application and disclaims any warranties, express or implied, to their use.

British Library Cataloguing in Publication Data

A catalogue record for this book is available from the British Library

ISBN: 978-1-80043-003-7 (Print)

ISBN: 978-1-80043-002-0 (Online)

ISBN: 978-1-80043-004-4 (Epub)

Dedication Page

To my parents and to Grigorios

Contents

List of Tables and Flowcharts xiii
Abbreviations xv
Foreword xvii
Chapter 1: Introduction 1
1.1. The Two Themes 1
1.2. Two Premises 3
1.2.1. Evidence-based Education 3
1.2.2. Skills-based Curricula 4
1.3. The Outline of this Book 6
Chapter 2: How to Conduct a Multidimensional Programme Evaluation 9
2.1. The Purpose of the Evaluation 9
2.2. Dimensions of Evaluation 10
2.2.1. Pedagogical Evaluation 11
2.2.2. Systematic Literature Review 11
2.2.3. Identify Ways to Fill in the Literature Gaps 11
2.2.4. Synthesising the Evidence 14
2.2.5. Offer Recommendations 15
Chapter 3: Philosophy for Children: Discussing About Pedagogy 17
3.1. Introduction 17
3.2. Philosophy for Children: A School-based Dialogic Intervention 17
3.3. The Purpose of the Evaluation 18
3.3.1. A Questioning Mind 19
3.3.2. Was There a Need for Evaluating Philosophy for Children? 20
3.4. Pedagogical Evaluation 22
3.4.1. Identify the Stance and Justify It 22
3.4.2. Investigate the Programme Rationale 23
3.4.3. Evaluate the Main Elements of the Programme: Learning and the Role of the Student and the Teacher 25
3.4.4. Examine Oppositional Views 29
3.4.5. Search for Inconsistencies and Areas for Improvement 32
3.4.6. Reach to a Conclusion 33
Chapter 4: Systematic Literature Review: Evaluating the Available Evidence 35
4.1. Conducting a Systematic Literature Review 35
4.1.1. Specify the Research Question and the Inclusion Criteria 35
4.1.2. Decide on the Way the Search Will Be Conducted and Identify the Relevant Literature 36
4.1.3. Evaluate the Quality of the Evidence 37
4.1.4. Calculating the Impact of the Programme 39
4.2. Philosophy for Children: A Systematic Literature Review 40
4.3. The Impact of Philosophy for Children on Students’ Skills 43
4.4. Limitations of the Review 46
4.5. A Summary of the Findings of the Systematic Literature Review 47
Chapter 5: Generating New Evidence 49
5.1. Conducting an Evaluation Study 50
5.1.1. Research Design 50
5.1.2. Pre-test and Post-test 52
5.1.3. Process Evaluation 52
5.1.4. Analysing the Data 53
5.1.5. Recognising and Reporting the Limitations 54
5.2. The Method of Conducting the Philosophy for Children Study 54
5.2.1. Intervention Group 55
5.2.2. Comparison Group 55
5.3. Response Rate and Missing Data 56
5.4. Process Evaluation and Analysis 56
5.5. Ethics 58
5.6. Limitations of the Research Design 58
Chapter 6: Evaluating the Impact on Thinking Skills: Working Definitions 61
6.1. Defining Critical Thinking 61
6.1.1. Lipman’s Definition of Critical Thinking 61
6.1.2. Critical Thinking as a Guide to Action 62
6.1.3. Critical Thinking or Critical Thinker? 64
6.1.4. Critical Thinking as Problem-solving 66
6.1.5. Critical Thinking as an Active Process 67
6.2. Two Important Debates 67
6.2.1. Is Critical Thinking Value-neutral? 67
6.2.2. Should Critical Thinking Be Considered a General or a Subject-specific Skill? 69
6.3. Critical Thinking: Working Definition 71
6.4. Defining Creativity 73
6.4.1. Person 74
6.4.2. Process 75
6.4.3. Product 76
6.4.4. Environment 77
6.5. Debates 77
6.5.1. Is Creativity Value-neutral? 77
6.5.2. Is Creativity a Domain-specific Skill? 78
6.6. Creativity: Working Definition 78
6.7. The Relationship Between Critical and Creative Thinking 80
6.8. Can Critical Thinking and Creativity Ever Be Improved? 81
6.9. Transparency 83
Chapter 7: Assessing Students’ Thinking Skills 85
7.1. Introduction 85
7.2. Critical Thinking Assessments 85
7.2.1. Halpern Critical Thinking Assessment (2010) 86
7.2.2. Cornell Critical Thinking Test (2005) 86
7.2.3. Ennis–Weir Critical Thinking Test Essay (1985) 87
7.2.4. Test of Appraising Observations (1984) 87
7.2.5. Watson Glaser Critical Thinking Appraisal (2002) 88
7.2.6. New Jersey Test of Reasoning Skills (1983) 89
7.3. Creativity Assessments 89
7.3.1. Multi-trial Creative Ideation 89
7.3.2. Latent Semantic Analysis 90
7.4. Measurement Tools for the Comparative Evaluation Study 90
7.5. Assessing Students’ Creativity 92
7.6. Marking Students’ Creativity 93
7.6.1. First Activity 94
7.6.2. Second Activity 96
7.6.3. Calculating the Overall Creativity Score 96
7.7. Designing the Critical Thinking Assessment 98
7.7.1. Purpose 99
7.7.2. Construct 99
7.7.3. The Form of the Assessment 99
7.7.4. Why Three Alternatives in the Multiple-choice Questions? 100
7.7.5. Guidelines for Constructing Good Multiple-choice Items 101
7.8. Sample Items of Critical Thinking Assessments 102
7.8.1. Evaluation of an Argument and Credibility of Sources 102
7.8.2. Deduction 104
7.8.3. Assumption Identification 105
7.9. Marking 106
7.10. Psychometric Properties 107
7.10.1. Reliability 107
7.10.2. Validity 108
7.11. Piloting of the Assessments Used in the Philosophy for Children Evaluation 110
Chapter 8: The Findings of the Quasi-experimental Study 111
8.1. The Impact of the Philosophy for Children Programme on Critical Thinking 111
8.2. The Impact on Different Critical Thinking Skills 112
8.3. The Impact of the Philosophy for Children Programme on Creativity 114
8.4. The Impact on Different Aspects of Creativity 115
8.5. A Summary of the Findings of the Quasi-experimental Study 117
Chapter 9: Secondary Data Analysis: Evaluating the Long-term Impact of the Programme 119
9.1. Secondary Data Analysis as an Evaluation Method 119
9.2. Cases 120
9.3. Missing Data 122
9.4. Analysis 123
9.5. Findings of the Secondary Data Analysis 123
9.5.1. Results: Impact on Attainment 124
9.5.2. Results: Impact on Disadvantaged Students’ Attainment 125
9.6. Discussion 125
Chapter 10: Combining the Evidence: Deciding About the Programme Effectiveness 127
10.1. Philosophy for Children: Synthesising the Evidence 128
10.2. Should the Programme Be Implemented? 133
Chapter 11: Recommendations: When and How Should the Programme Be Implemented in the Classrooms? 135
11.1. Recommendations for Practitioners 135
11.1.1. Thinking Skills 136
11.1.2. Social Skills 136
11.1.3. Self-esteem and Self-confidence 138
11.1.4. Well-being 139
11.1.5. Recommendations on Programme Fidelity 140
11.1.6. Time Allocated 140
11.2. Closing the Gap Between Advantaged and Disadvantaged Students: Recommendations for Policy Makers 141
11.3. Recommendations for Teacher Education 141
11.4. Recommendations for Researchers 142
Chapter 12: Concluding Thoughts 143
12.1. Future Directions 143
12.2. Teaching and Assessing Skills in Schools 144
12.2.1. Critical and Creative Thinking 145
12.2.2. How Can Schooling Support the Development of Students’ Thinking Skills? 146
12.2.3. Assessing Creativity and Critical Thinking 147
12.3. Final Thoughts 148
References 149
Index 163

List of Tables and Flowcharts

Tables
Table 1. Rating Scale to Evaluate the Trustworthiness of a Study Which Evaluates the Impact of a School-based Programme. 38
Table 2. Quality of Research Design and Reporting of the Studies Included in the Systematic Literature Review. 41
Table 3. Quality of Studies in Relation to the Calculated Effect Sizes (P4C Impact). 42
Table 4. Matrix with Inter-item Correlations for the Pre-test. 97
Table 5. Matrix with Inter-item Correlations for the Post-test. 97
Table 6. Simple Version of the Test Blueprint for the Critical Thinking Assessments. 102
Table 7. Impact of P4C on Critical Thinking. 112
Table 8. Impact of P4C on Students’ Critical Thinking Skills. 113
Table 9. Impact of P4C on Creativity. 114
Table 10. Impact of P4C on Students’ Creative Thinking Skills. 115
Table 11. Number of Students Eligible for FSM and Non-FSM Per Group. 122
Table 12. Frequency of Pupils’ Assessment Data in Maintenance Schools. 123
Table 13. Impact of P4C on Attainment. 124
Table 14. The Synthesis of Evidence. 129
Table 15. Weighing the Evidence. 134
Flowcharts
Flowchart 1. Participant Flow Chart for the Quasi-experimental Study. 57
Flowchart 2. Participant Flow Chart for the Secondary Data Analysis. 122

Abbreviations

P4C Philosophy for Children
RCT(s) Randomised Controlled Trial(s)
TTCT Torrance Tests of Creative Thinking
HCTA Halpern Critical Thinking Assessment
CCTT Cornell Critical Thinking Test
SD Standard Deviation
NPD National Pupil Database
FSM Free School Meals

Foreword

Professor Chris Brown, Durham University School of Education

The world we inhabit, COVID-19 aside, has had very mixed fortunes over the last decade. On the one hand, technology has enabled us to achieve things our ancestors would never have believed possible: redefining the way people communicate, collaborate, shop, travel, read, research, watch films, gather information, book holidays, bank and so much more (Greengard, 2015). Digital photography and social media have enabled us to capture, represent and share the world in previously unimaginable ways, while tools such as 3D printers allow us to make real our designs for anything, from sculptures to bridges, at the touch of a button. In terms of the economy, artificial intelligence (AI) and mass automation have been transformative, with much routine and low skilled work now undertaken by robots or algorithms. This use of AI is set to continue, with most commentators agreeing that AI will eventually take over many of the tasks machines can perform equally or better than humans: everything from processing insurance claims to space exploration. And this is problematic: with pre-COVID estimates indicating that, over the next 20 years, some 47% of jobs in the United States and 54% of those in Europe will be lost to machines (Bregman, 2018; Frey & Osbourne, 2013).

But what separates humans from machines is our ability to engage in creative thinking – which along with its alter ego, critical problem solving – is about constructing more or less novel ideas, objects or even worlds (du Sautoy, 2019). It is ‘imaginative activity fashioned so as to produce outcomes that are both original and of value’ (National Advisory Committee on Creative and Cultural Education (NACCCE), 1999): in other words, activity designed to produce ideas that are not only innovative, but also fit for purpose (Durham Commission on Creativity and Education, 2019). What is novel, fit for purpose and indeed somehow satisfying, will depend on different domains: in what Victorian designer William Morris describes as the lesser arts (such as interior decoration), there can be a strong emphasis on novelty. In fields such as architecture, fitness for purpose is generally likely to receive more attention. In all cases, however, our acts of creation elevate, expand and transform what it means to be human.

But if it’s creativity that is keeping us one step ahead of the machines, then education systems globally now need to be focussed on actively supporting future citizens to be able to collaboratively engage in critical and creative thought. In other words, we need education systems to now arm students with the capacities required to explore, experiment, try and re-work, make and re-make, explore and value difference, overcome obstacles and develop and apply knowledge (Newton & Newton, 2018). But we still need practical suggestions for how to make this happen. Also, to have the confidence that if we are to embrace approaches to creative problem solving, that these will make a concrete and positive difference. With this extraordinary new book, Dr Rania Ventista has managed to meet these twin goals. Taking us on a powerful journey of exploration, not only does Dr Ventista showcase, with some considerable skill, how to evaluate educational interventions effectively; she also illustrates in detail why one intervention in particular – Philosophy for Children – matters.

The results are a framework that can and should be used by teachers and school leaders when understanding how to allocate scarce resources. But at the same time, how Philosophy for Children can enhance creative problem solving and why educators should be embracing this programme to arm their students with the skills they need to navigate the perils and pitfalls the twenty-first century holds. With this, her first book, Dr Ventista has announced to the world her skills as a scholar and leading thinker in this field. I can’t wait for what’s to come!

References

Bregman, R. (2018). Utopia for realists and how we can get there. London: Bloomsbury.

du Sautoy, M. (2019). The creativity code: How AI is learning to write, paint and think. London: 4th estate.

Durham Commission on Creativity and Education. (2019). Durham Commission on Creativity and Education. Retrieved from https://www.artscouncil.org.uk/sites/default/files/download-file/Durham_Commission_on_Creativity_04112019_0.pdf. Accessed on March 5, 2021.

Frey, C., & Osborne, M. (2013). The future of employment: How susceptible are jobs to computerisation. Retrieved from https://www.oxfordmartin.ox.ac.uk/downloads/academic/The_Future_of_Employment.pdf

Greengard, S. (2015). The internet of things. Cambridge, MA: MIT Press.

National Advisory Committee on Creative and Cultural Education (NACCCE) (1999). All our futures: Creativity and cultural education. London: DfEE.

Newton, L., & Newton, D. (2018). Making purposive thought productive. Ulm: The International Centre for Innovation in Education.