Search results

This policy brief, the second AERI-NEPC eBrief in the series “Understanding validity issues around the world”, focuses on validity as it applies to test-based models of evaluation employed for schools, instructional programs, and teachers around the world. It discusses validity issues that could arise when data from student achievement test administrations and other sources are used for conducting personnel appraisals, program evaluations, or for external accountability purposes, suggesting solutions and recommendations for improving validity in such applications of test-based information.

This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. It begins by summarizing perspectives of an invited expert panel on the topic. To that synthesis, the authors add their own analysis of key issues. They conclude by offering recommendations for test developers and test users.

The authors conclude that systematic improvement and transformation of schools depends on thoughtfully conceptualizing, implementing, and using data from testing and broad-based evaluation systems that incorporate multiple kinds of evidence. Evaluation systems that are valid and fair to students, teachers and education leaders need all three of the following: assessment resources and training for all participants and evaluation users; knowledgeable staff to continuously monitor processes and use assessment results appropriately to improve teaching and learning activities; and a strengths-based approach to make improvements to the education system based on relevant data and reports (as opposed to a deficits-based one in which blame or punishment is leveled at individuals or groups of workers when gaps in performance are observed).

To improve validity in interpretations of results from test-based teacher and school evaluation models, the authors provide recommendations for measurement and evaluation specialists as well as for educators, policy makers, and public users of data. Standardized test use in formative and more “high stakes” educational accountability contexts is rapidly spreading to various regions of the world. This eBrief shows that understandings of validity are still uneven among key stakeholders. By translating complex information pertinent to current validity issues, this policy brief attempts to address this need, and also bridge knowledge and communications gaps among different constituencies.

What do principals look for when hiring teachers? The purpose of this paper is to extend the knowledge concerning what aspects of teacher quality are in demand among the individuals who administer schools and make hiring decisions.

Rather than employing interviews or surveys, the authors utilized a conjoint instrument that assembled teacher characteristics into fictitious applicant profiles. Participating North Carolina public school principals (n = 467) then chose among the computer-generated options and regression analysis allowed the authors to identify preferences in the aggregate.

Principals in this study preferred applicants with classroom experience, but those with 15 years were no more preferred than those with 5. They also preferred applicants with more education, but an advanced degree was no more preferred than a bachelor’s from a highly selective institution. Preference for teachers who are committed to state standards varied with schools’ performance on state tests.

Conjoint analysis is a useful tool for measuring preferences but is underutilized in research on education administration. This paper contributes not only to the body of knowledge about school principal behavior but also to the field’s familiarity of research techniques.

Policymakers have focused on improving STEM outcomes for US high school students for over 50 years. Much of this focus has centered on improving the quality of STEM teachers, particularly in poor and minority schools. Few, if any, of these efforts have considered the importance of the content knowledge of those providing instructional leadership in schools – namely, principals and assistant principals. This chapter examines the percentage of school leaders with teacher certification in mathematics or science and the degree to which teacher and school leader turnover interrupts the leadership–teacher relationships. The study concludes relatively few school leaders have the content knowledge to provide deep instructional leadership. Moreover, the study finds combined teacher and school leader turnover greatly diminishes the sustained relationships between instructional leaders and teachers, particularly in lower-performing schools.

Though test-based accountability policies seek to redress educational inequities, their underlying theories of action treat inequality as a technical problem rather than a political one: data point educators toward ameliorative actions without forcing them to confront systemic inequities that contribute to achievement disparities. To highlight the problematic nature of this tension, the purpose of this paper is to identify key problems with the techno-rational logic of accountability policies and reflect on the ways in which they influence teachers’ data-use practices.

This paper illustrates the data use practices of a workgroup of sixth-grade math educators. Their meeting represents a “best case” of commonplace practice: during a full-day professional development session, they used data from a standardized district benchmark assessment with support from an expert instructional leader. This sociolinguistic analysis examines episodes of data reasoning to understand the links between the educators’ interpretations and instructional decisions.

This paper identifies three primary issues with test-based accountability policies: reducing complex constructs to quantitative variables, valuing remediation over instructional improvement, and enacting faith in instrument validity. At the same time, possibilities for equitable instruction were foreclosed, as teachers analyzed data in ways that gave little consideration of students’ cultural identities or funds of knowledge.

Test-based accountability policies do not compel educators to use data to address the deeper issues of equity, thereby inadvertently reinforcing biased systems and positioning students from marginalized backgrounds at an educational disadvantage.

This paper fulfills a need to critically examine the ways in which test-based accountability policies influence educators’ data-use practices.

This AERI-NEPC eBrief, the fourth in a series titled “Understanding validity issues around the world”, looks closely at issues surrounding the validity of test-based actions in educational accountability and school improvement contexts. The specific discussions here focus testing issues in the US. However, the general principles underlying appropriate and inappropriate test use in school reform and high stakes public accountability settings are applicable in both domestic and international settings. This paper aims to present the issues.

This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. It begins by summarizing perspectives of an invited expert panel on the topic. To that synthesis, the authors add their own analysis of key issues. They conclude by offering recommendations for test developers and test users.

The authors conclude that recurring validity issues arise with tests used in school reform and public accountability contexts, because the external tests tend to be employed as policy instruments to drive reforms in schools, with unrealistic timelines and inadequate resources. To reconcile the validity issues with respect to educational assessment and forge a coherent understanding of validity among multiple public users with different agendas, the authors offer several recommendations, such as: adopt an integrated approach to develop content and standards of proficiency that represent a range of cognitive processes; support studies to examine validity of assessments and the effects of decisions taken with assessment data before results are fed into high stakes accountability-related actions that affect teachers, leaders or schools; align standards, curricula, instruction, assessment, and professional development efforts in schools to maximize success; increase capacity-building efforts to help teachers, administrators, policy makers, and other groups of test users learn more about assessments, particularly, about appropriate interpretation and use of assessment data and reports.

Baker points out that in response to growing demands of reformers and policy-makers for more frequent and rigorous testing programs in US public education, results from a single test tend to get used to meet a variety of public education needs today (e.g. school accountability, school improvement, teacher evaluation, and measurement of student performance). While this may simply be a way to make things more cost-efficient and reduce the extent of student testing in schools, a consequence is inappropriate test use that threatens validity in practice settings. This policy brief confronts this recurring validity challenge and offers recommendations to address the issues.

The purpose of this paper is to examine how test‐based accountability has influenced school and district practices and explore how states and districts might consider creating expanded systems of measures to address the shortcomings of traditional accountability. It provides research‐based guidance for entities that are developing or adopting new measures of school performance.

The study relies on literature review, consultation with expert advisers, review of state and district documentation, and semi‐structured interviews with staff at state and local education agencies and research institutions.

The research shows mixed effects of test‐based accountability on student achievement and demonstrates that teachers and administrators change their practices in ways that respond to the incentives provided by the system. The review of state and district measurement systems shows widespread use of additional measures of constructs, such as school climate and college readiness.

There is a clear need for additional research on the short‐ and long‐term effects of expanded systems of measures. In particular, currently little is known about how the inclusion of input and process measures influences educators’ practices or student outcomes.

The research suggests several practical steps that can be taken to promote effective systems of measurement, including providing supports for high‐quality teaching to accompany new measures, offering flexibility to respond to local needs, and conducting validity studies that address the various purposes of the measures.

The paper provides new information about how states and districts are expanding their systems of measures for various purposes, and informs accountability policy by highlighting the benefits and limitations of current outcomes‐based approaches to accountability and by clarifying the trade‐offs and decisions that should be considered.

The purpose of this paper is to investigate how principals and school supervisory authorities understand and use feedback from mandatory proficiency tests (VERA) in the low-stakes context of Germany. For the analysis, the authors refer to a theoretical model of schools that differentiates between Autonomous and Managed Professional Organisations (Thiel, 2008a).

The empirical basis of the qualitative research are contrastive case studies which are focusing on individual schools and on school supervisory authorities. The selection of cases was oriented on Theoretical Sampling according to Glaser and Strauss (2005). For the analysis, the authors used a sample of upper track secondary schools (Gymnasien), four from Berlin and five each from Thuringia, Brandenburg and Baden-Wuerttemberg. In total, the authors conducted 229 structured interviews over two periods in 19 schools with different protagonists on all levels of the educational system (principals, heads of subject departments, teachers and school supervisory officials). The interview data were descriptively analysed according to procedures of qualitative content analyses (Mayring, 2010).

The analyses show a clear tendency in the direction of the Autonomous Professional Organisation within the context of VERA. However, some principals reported activities according to a Managed Professional Organisation. The traditional decoupling remains and the supervisory authorities retain their picture of the individual school as an Autonomous Professional Organisation. Both levels have a major deficit in a competent use of VERA and lack profound experience with accountability and evaluation processes.

The sampling has certain restraints: schools of a particular type, few schools within one state, four of 16 states.

Even though German test-based school reforms have been in progress for one decade, systematic analyses of the reform’s relevance for leadership and for school supervisory authorities’ actions in a low-stakes context have not been conducted yet. The analysis meets this lack of research with an explorative reconstruction of principal leadership within the context of test-based school reform as well as the corresponding school supervisory officials.

The prevailing theory of action underlying accountability is that holding schools and students accountable will increase educational output. While accountability's theory of action intuitively seemed plausible, at the point of No Child Left Behind's national implementation, little empirical research was available to either support or critique accountability claims or to predict the long‐term impact of accountability systems on the success of at‐risk students and the schools that served them. The purpose of this paper is to analyze the work and perceptions of school teachers and leaders as they seek to meet the requirements of educational accountability.

Interviews with 89 administrators, staff and teachers revealed a variety of methods utilized to manage risks associated with low test scores and accountability ratings.

The findings reported in this paper challenge the proposition that accountability improves the educational outcomes of at‐risk students and indicates that low‐performing Texas high schools, when faced with the press of accountability, tend to mirror corporate risk management processes, with unintended consequences for at‐risk students. Low‐scoring at‐risk students were often viewed as liabilities by school personnel who, in their scramble to meet testing thresholds and accountability goals, were at‐risk student averse – implementing practices designed to “force kids out of school.”

In this paper, the authors use theory and research on risk management to analyze the work and perceptions of school teachers and leaders as they seek to meet the requirements of educational accountability. This paper is among the first to use this particular perspective to conceptualize and understand the practices of educational organizations with regards to the treatment of at‐risk students attending low‐performing high schools in the midst of accountability.

The purpose of this paper is to analyze the district effectiveness literature. It begins by summarizing the school effectiveness research, the correlates of effective schools, and the conceptual and methodological characteristics of this field. It then describes the findings from a review of 50 studies of district effectiveness, the most frequently identified correlates of effective districts, and the conceptual and methodological features of this research. From there, it compares and contrasts the two fields, paying attention to the ways in which they frame notions of success, purposes of education, the contextualized nature of school performance, and theoretical explanations for student success.

Data sources for this literature review included 50 primary documents on district effectiveness. The studies were bound to those that presented the original results from investigations of the relationship between district‐level policies, routines, behaviors, or other characteristics and classroom‐level outcomes.

Several themes run through the literature on district effectiveness. These include findings that standards‐aligned curricula, coherent organizational structures, strong instructional leadership, frequent monitoring and evaluation, and focused professional learning lead to higher test scores. Most of these investigations are framed from technical perspectives that explore the relationship between organizational regulations and improved test performance. Less common are inquiries about the socio‐political and normative forces that shape districts’ improvement experiences. One consequence of this technical focus is that the field of district effectiveness has come to share several of the conceptual and methodological properties that characterized the former school‐level research.

The article concludes by discussing the implications for the growing volume of district‐level research on educational leadership, district improvement, and educational equity.

This article details the ways in which a sharp focus on questions of what works in the district effectiveness literature has deepened researchers’ and practitioners’ knowledge of the specific mechanisms that may produce more desirable results in test performance, yet these questions alone, decoupled from corresponding inquiries about the complex, highly contextualized character of higher or lower scoring districts, leave researchers and practitioners vulnerable to the same scholarly and practical pitfalls of their predecessors.