Data use for equity: implications for teaching, leadership, and policy

Citation

Datnow, A., Greene, J.C. and Gannon-Slater, N. (2017), "Data use for equity: implications for teaching, leadership, and policy", Journal of Educational Administration, Vol. 55 No. 4, pp. 354-360. https://doi.org/10.1108/JEA-04-2017-0040

Data use for equity: implications for teaching, leadership, and policy

Introduction to data use and equity

The articles in this special issue provide cutting-edge research knowledge on the intersection of two important policy priorities in the field of education: data-driven decision making and equity. Both of these priorities have critical implications for the work of teachers and school leaders, for school improvement initiatives, and ultimately for student learning. Conceivably, data use could help accomplish goals of equity, and equity could drive data use efforts. However, up to now, the field has had little knowledge about how equity and data use come together in the process of educational improvement. This issue is aimed at addressing this knowledge gap.

Providing an overview of data-driven decision making in education is a logical place to start in this effort. First, what is data-driven decision making? Data-driven decision making is the notion that important decisions will be anchored in data, rather than simply being based on hunches about the right course of action. Originally deriving from the field of management, the verve for data use is prevalent not just in education but across a wide range of sectors. Organizations are expected to be data-driven, as are the individual decision makers within them. What “data” means varies across contexts, but it typically refers to systematically gathered information. Over time, researchers have questioned the notion that data in fact drive (Dowd, 2005) and have argued that a more useful conception is that data inform decision making (Datnow and Park, 2014). We clearly understand the limits of the term data-driven, but we use the terms data-driven and data-informed somewhat interchangeably here.

For more than a decade, data-driven decision making has been a prominent feature of educational reform agendas across the globe, including in the USA, Canada, Spain, The Netherlands, South Africa, and New Zealand, among many other countries. Data-driven decision making in education “refers to teachers, principals, and administrators systematically collecting and analyzing various types of data […]. To guide a range of decisions to help improve the success of students in schools (Marsh et al., 2006, p. 1).

In education, Lai and Schildkamp (2016) explain that data use derives from two often-competing agendas. On the one hand, data use is promoted as part of an external accountability framework. On the other hand, data use is integral to an agenda of teacher inquiry, contributing to internal accountability. In the first case, large scale assessment data dominate, and in the second, a much wider range of data is used to inform instructional decision making. Lai and Schildkamp explain that teachers often have to balance these two competing agendas simultaneously.

While agendas for data use vary, the theory of action underlying these data use efforts in education has similar contours across contexts (Ikemoto and Marsh, 2007). The common idea is that the examination of data by educators will lead to decision making that is better informed and ultimately more attuned to student and organizational needs. The process for data use – at least in typical form – begins with setting a goal, gathering data, analyzing data, using data to inform a plan of action, evaluating the results, and repeating the cycle with refinements.

The part of the cycle that educators tend to struggle with the most is using data to inform action. This is due to several reasons. First, time constraints often mean that educators spend more time gathering and examining the data than they do planning on the basis of data. They essentially run out of time and action plans tend to be cursory in nature. Moreover, at the teacher level, taking action on the basis of data requires an expansive tool box of strategies to meet instructional goals in new ways. Teachers sometimes feel they are lacking capacity in this area, and when this occurs, data-informed instructional decision making can fall flat.

Many district and school leaders have organized teachers into professional learning communities to build in order for teachers to jointly solve problems and share instructional strategies for addressing students’ needs. Whether or not professional learning communities help schools realize the goals of data use depends a great deal on the nature of the inquiry around the data (Daly, 2012; Horn et al., 2015). Teachers are recognized as important sense makers who bring their professional judgment to bear in the process of examining and acting upon data (Bertrand and Marsh, 2015). Therefore, no decision is actually fully data-driven; rather, people filter data through their own lenses and experiences and chart a course of action accordingly. In this way, teachers’ intuition does come into play in important ways in the data use process.

Educational leaders play a critical role in the data use process (Datnow and Park, 2014; Halverson et al., 2007; Knapp et al., 2007). In fact, it is rare to find a data use effort in education that did not begin with leadership. Leaders at the school and district level establish cultures and structures to support data use. The way in which a leader frames data use efforts is critical and influences which data are prioritized, how data are used and for what purposes, teachers’ levels of trust and comfort in sharing data with each other, and other important factors that are enormously influential for school improvement (Park et al., 2013). Leaders also play an important role in modeling thoughtful use of data, rather than data-driven decision making that is accountability-driven and oriented toward quick fixes.

At the same time that data use is a ubiquitous part of policy agendas, equity continues to be a priority goal for most contemporary educational initiatives. Equity is defined in various ways in our field. The Glossary of Educational Reform (2016) defines equity in terms of the principle of fairness, further explaining that programs or policies are inequitable when they lead to unequal outcomes for groups of students. Reforms aimed at equity typically seek to identify disparities and then remedy inequities through a variety of means. OECD (2008) also defines equity in terms of fairness, as well as through inclusion. That is, personal and social obstacles should not be a barrier to education, and every child must be ensured a basic minimum standard of education. Whereas OECD focuses on a basic minimum standard, Pollock (2017) defines equity as supporting the full human talent development of every student and all groups of students. We believe that supporting all students to reach their full potential is an important goal of equity efforts in schools. How can data use play a role in this?

Understanding how data use contributes to equity within the broader policy and reform context, both in the USA and internationally, is extremely important. Although few studies to date (for an exception see Bertrand and Marsh, 2015) focus specifically on the equity-data use intersection, a number of prior studies have uncovered findings that reveal that potentially significant equity implications. For example, Lachat and Smith (2005) found that using data can help address false assumptions about the root causes of student achievement patterns. In this study, a careful examination of data helped teachers realize that low student achievement among specific groups of students was not related to attendance patterns, as originally thought, and in fact also had to do with the quality of instruction. However, when data are examined in a more superficial manner, teachers may not have an opportunity to dig deeply into these kinds of questions (Hoover and Abrams, 2013) and assumptions can go unexamined. Even more troubling, Bertrand and Marsh’s (2015) study found that when educators used student characteristics as an explanation for results, they deflected attention away from their own practice and reinforced a culture of low expectations.

The relationship between data use, accountability, and equity is also important to consider. When data use is inextricably connected with high stakes accountability systems, goals for equity may be compromised as educators tend to focus remediation efforts on a small group of students on the cusp of proficiency or what some call “the bubble” (Booher-Jennings, 2005). On the other hand, when data use is a priority of national or local governments, but no national accountability system exists, educators are freer to orient data use endeavors around pressing problems in their local context. Similarly, in countries without school districts or local education authorities and where schools operate more autonomously, the supports and pressures for data use will vary more widely. The presence or absence of national curricula also plays a role in data use for equity.

Equity questions also arise with respect to how data are shared and used by students. While conceivably sharing data with students can help them better chart their own plans for improvement, Neuman’s (2016) study found that data use can further marginalize students when students are reminded that they are failures by color-coded spreadsheets that display their test scores publicly in the classroom. All of the aforementioned studies point to the fact that whether and how equity is supported in data use efforts depends a great deal on the day-to-day decisions of educators in local contexts. However, there is still much to learn.

The articles in this special issue

With a decade of data use policies and practices behind us, what indeed is the relationship between data use and equity? The articles in this special issue address this question from a variety of different perspectives. These five rigorous, empirical articles emerge from studies that are part of the Spencer Foundation’s Data Use and Educational Improvement initiative. Focusing explicitly on “real life” data use, the primary objective of these articles is to explore how data use policies and practices influence equitable learning opportunities for students, as well as challenge or reinforce existing equity stances of teachers and leaders. Reporting findings on data use-in-practice in a variety of contexts, the authors portray how teachers, coaches, and principals take up data use policies in distinct ways and, in turn, how these diverse interpretations yield a range of intended and unintended consequences of data use policies for equity. Taken together, these articles help connect microanalyses of data use efforts with macro questions about equity in schools and society.

The articles are written by a diverse group of US-based authors, including senior and more junior scholars representing nine institutions, who have prior experience as educational leaders, teachers, and evaluators and who also currently train teachers and leaders. The research was in contexts with some similar policy parameters and educational rationales for data use initiatives. Thus, our work collectively represents one main national context for data use, with local variation at the state and district levels. While the work reported herein is US based, we can extrapolate lessons for data use policies in both US and international contexts.

Each article is marked by rich methodology, profound research questions, and commitment to drawing clear, actionable implications for data use policies and practice. This collection of studies relies primarily on qualitative methods, as these methods are best suited for explaining a phenomenon as situated in particular contexts. Understanding data use for instructional decision making requires close investigation into how educators engage with data so that we can find out why it fosters positive outcomes in some places and not others (Coburn and Turner, 2012). As Little (2012) argues, we need more studies that either “zoom in” on teachers’ daily and weekly activities around data, or those that “zoom out” to address how data use fits in within a larger context of teachers’ work. The articles in this issue do both, with a specific focus on the intersection between data use and equity. And, while all the articles rely on interpretive, qualitative designs, the methodologies of these range from ethnographies to case studies to mixed methods studies that integrate data from both qualitative and quantitative methods.

In sum, the research studies presented are diverse yet share common aims: to uncover how data use works in practice, to present equity-focused implications from the findings, and to draw out practical implications for leadership and school improvement.

The first article by Nora Gannon-Slater, Priya Goel, Hope Crenshaw, Margaret Evans, Jennifer Greene, and Thomas Schwandt examines key differences and missed opportunities for equity-focused data use under varying organizational cultures. As part of a larger study on data use in small urban schools, the researchers use a comparative observational case study of grade-level teams in two schools in one district. In this district, data use was an important dimension of system-wide equity reforms and included policies for professional learning communities and Response to Intervention. Data talk, which provided both a holistic and nuanced account of data use practices in grade-level meetings, revealed distinct macro data use cultures of accountability and organizational learning that strongly influenced practice. Despite these differences, there was little attention to the explicit issues of equity raised by the district. The results suggest that opportunities for equity-oriented data use are less prevalent in accountability cultures of data use, but organizational cultures of data use are only necessary but not sufficient for equity-minded conversations around student performance. The authors describe how school leaders who advocate that equity importantly guides data use routines must create additional contexts surrounding data use and anticipate how cultures of accountability or organizational learning “show up” in data use conversations in order to be better prepared to redirect teachers’ interpretations of data and clarify expectations of equity reform initiatives.

Alice Huguet, Caitlin C. Farrell, and Julie A. Marsh use a comparative case study of two schools within one district to illustrate how school leadership approaches to creating, implementing and maintaining tools and routines advance or constrain equity-focused data use. The research draws on mixed methods, including surveys, interviews, focus groups, document review, and participant and field observations of two middle schools serving large numbers of low-income and minority students. In their study of the district’s investment in professional learning communities to support teacher use of literacy data, they find diverging leaders, one principal with a “light touch” and one with a “heavy hand” in creating the structures and supports for data use. Their study reinforces the fact that protecting teacher collaborative time is a critical enabler for teacher data use. However, “[l]ess obvious is the finding that the degree to which a principal controls a PLC’s tools and routines may influence teachers’ perceptions of the PLC’s usefulness” (p. 376), where a “heavy hand” described in part as less teacher control to structure their time and co-construct tools decreased teacher buy-in. On the other hand, they suggest a “light touch” may be better for “creating conditions under which teachers may have had a greater opportunity to engage meaningfully around data […] a necessary, though not sufficient, condition for productive data use” (p. 376). The broader conclusion is the important role principals appear to play in varied and uneven implementation and the consequences this has for equitable access to quality inputs in educational environments.

The article by Vicki Park, Elise St John, Amanda Datnow, and Bailey Choi reports key findings from a study of three schools’ “classroom placement routines” – the processes by which students are placed in particular classrooms with particular teachers – and the role of data therein. Drawing from a larger set of case studies on data use, they use teacher and leader interviews, observations of placement meetings, and document review to study classroom placement routines and decisions from three elementary schools. The researchers investigated how ostensive aspects of the routine “provided the general template and resources in action,” where the performative aspects of the routine illustrated the “multiple and varied nature of how routines are enacted in practice” (p. 390), both of which can illustrate patterns of equity and inequity.

Of major influence across schools was the need to “balance” various student characteristics (for example, gender and academic strengths) across classrooms, yet also keep friends together, as well as some students with similar special needs. In striving toward this goal, they found that “data were rarely the center of discussions but instead were strategically deployed to play confirming or disconfirming roles, to highlight discrepancies […] to speak specifically about student skills […] and to contextualize students’ performances and abilities, p. XX.” The study highlights how assumptions about student backgrounds and teacher expectations play out in placement decisions and the potential power and influence that leaders have in maintaining or disrupting these assumptions in practice.

Brett Garner, Jennifer Kahn Thorne, and Ilana Seidel Horn use an in-depth study of teacher data use to illustrate the consequences of prevalent educational policies of accountability for equity. Drawing from a larger study on instructional improvement in middle school mathematics instruction, these researchers adopted a critical lens through which to analyze, interpret, and critique a day-long workgroup discussion of benchmark assessments by a trio of middle school mathematics teachers. In their analyses, they draw on two visions of data use for equity: the techno-rational vision, which “embraces neoliberal notions of meritocracy” (p. 407), in comparison to the culturally responsive vision, which interprets test performance through socio-contextual and cultural lenses.

A primary finding from this analysis was that teacher talk about these benchmark data was framed and supported by techno-rational logics that did not question or challenge either the veracity of the data, or the teachers’ own interpretations of the data as weaknesses in student mastery. This finding is concretely and persuasively illustrated in excerpts from the teachers’ data use conversations. Subsequently, interpretations of data as a function of misdirected or inadequate instruction did not surface and follow-up actions emphasized re-teaching rather than instructional change or improvement. In these ways, data interpretation and use become a mere technical enterprise that supports the current educational system of accountability, rather than a meaningful engagement with the whole child and his/her unique cultural profile of strengths and limitations. The researchers’ critical lens successfully illustrates a “misalignment of the techno-rational vision […] and the culturally responsive visions” (p. 407) for equity.

Melissa Braaten, Chris Bradford, Kathryn L. Kirchgasler, and Sadie Fox Barocas narrate the pedagogical and equity tensions invoked by a mandated data-use accountability initiative. A two-year ethnographic study of science teachers demonstrated how a district initiative came to supplant teachers’ commitments to active student exploration and problem solving and to equity of student learning opportunities, experiences, and accomplishments. These authors argue that critical to the power of the district’s “data-centric accountability efforts” was its development as a “bundled” initiative. “When leaders and teachers enact initiatives rooted in data-centric policies, a host of [other] messages, tools, goals and strategies are bundled together. Bundles are [powerful] sites for framing and sensemaking in organizations” (p. 427). This district’s bundled set of data use initiatives foregrounded disparate test scores as the inequity problem facing educators. Over time, the focus on test scores gave way to dissonance with science teachers’ commitments to active learning, student-centered pedagogy, and equity of student learning. In short, the bundling mechanism empowered the data use initiative at the cost of slowly eroding teacher agency, use of professional judgment in pedagogical and curricula decisions, and, concomitantly, equitable opportunities for student meaningful learning about science.

The articles in this issue capitalize on the increasing prevalence of data use routines, build on the paucity of research that examines practices “in situ,” and are anchored in the legitimate belief that effective practices of data use will improve instruction and outcomes for students. Taken together, the articles provide theoretically and empirically informed understandings of endemic tensions that exist in leadership roles and responsibilities in cultivating data use practices in schools. Our intention was to offer several considerations for leaders in their design, implementation and monitoring of data use initiatives, particular tools and routines. The outcome would appear a set of necessary but insufficient conditions for data use to serve equity aims, and we hope they fuel many conversations and investigations into how these conditions can be created and sustained in current educational environments.