Search results

Fabricated data jeopardize the reliability of large-scale population surveys and reduce the comparability of such efforts by destroying the linkage between data and measurement constructs. Such data result in the loss of comparability across participating countries and, in the case of cyclical surveys, between past and present surveys. This paper aims to describe how data fabrication can be understood in the context of the complex processes involved in the collection, handling, submission and analysis of large-scale assessment data. The actors involved in those processes, and their possible motivations for data fabrication, are also elaborated.

Computer-based assessments produce new types of information that enable us to detect the possibility of data fabrication, and therefore the need for further investigation and analysis. The paper presents three examples that illustrate how data fabrication was identified and documented in the Programme for the International Assessment of Adult Competencies (PIAAC) and the Programme for International Student Assessment (PISA) and discusses the resulting remediation efforts.

For two countries that participated in the first round of PIAAC, the data showed a subset of interviewers who handled many more cases than others. In Case 1, the average proficiency for respondents in those interviewers’ caseloads was much higher than expected and included many duplicate response patterns. In Case 2, anomalous response patterns were identified. Case 3 presents findings based on data analyses for one PISA country, where results for human-coded responses were shown to be highly inflated compared to past results.

This paper shows how new sources of data, such as timing information collected in computer-based assessments, can be combined with other traditional sources to detect fabrication.

Large-scale assessment has been used in many education systems as an instrument to evaluate educational performance nationally. This practice is based on the concept of epistemic governance which encourages school accountability. This study aims to explore teachers' perspectives regarding the value and uses of national large-scale assessment (NLSA), highlighting its relevance across contexts.

Using qualitative data, this paper presents the case of the Israeli NLSA tests – the Meitzav, while examining the perceptions and actions in which teachers engage to follow-up on the test results, and the extent to which they implement pedagogical change in light of the test results.

The findings showed that teachers tend to use the NLSA test results as a pedagogical tool to improve learning processes to a limited extent. They concede that most activity involving the tests at the school and class levels is dedicated to preparation and not to pedagogical change. Some explanations are suggested.

This paper discusses the theoretical and practical implications of the NLSA testing regime for the school, curriculum and pedagogy.

This policy brief, the third in the AERI-NEPC eBrief series “Understanding validity issues around the world”, discusses validity issues surrounding International Large Scale Assessment (ILSA) programs. ILSA programs, such as the well-known Programme of International Student Assessment (PISA) and the Trends in International Mathematics and Science Study (TIMSS), are rapidly expanding around the world today. In this eBrief, the authors examine what “validity” means when applied to published results and reports of programs like the PISA.

This policy brief is based on a synthesis of conference proceedings and review of selected pieces of extant literature. It begins by summarizing perspectives of an invited expert panel on the topic. To that synthesis, the authors add their own analysis of key issues. They conclude by offering recommendations for test developers and test users.

ILSA programs and tests, while offering valuable information, should be read and used cautiously and in context. All parties need to be on the same page to maximize valid use of ILSA results, to obtain the greatest educational and social benefits, and to minimize negative consequences. The authors propose several recommendations for test makers and ILSA program leaders, and ILSA users. To ILSA leaders and researchers: provide more cautionary information about how to correctly interpret the ILSA results, particularly country rankings, given contextual differences among nations. Provide continuing psychometric or research resources so as to address or reduce various sources of error in reports. Encourage policy makers in different nations to share the responsibility for ensuring more contextualized (and valid) interpretations of ILSA reports and subsequent policy development. Raise awareness among policy makers to look beyond simple rankings and pay more attention to inter-country differences. For consumers of ILSA results and reports: read the fine print, not just the country rankings, to interpret ILSA results correctly in particular regions/nations. When looking to high-ranking countries as role models, be sure to consider the “whole picture”. Use ILSA data as complements to other national- and state-level educational assessments to better gauge the status of the country's education system and subsequent policy directions.

By translating complex information on validity issues with all concerned ILSA stakeholders in mind, this policy brief will improve uses and applications of ILSA information in national and regional policy contexts.

This paper aims to provide insight into the use of a standard for condition assessment.

The paper is based on a literature review, an analysis of the development, content and practical use of the Dutch Standard for Condition Assessment of Buildings, and the findings of several research projects about condition assessment and maintenance planning by Dutch housing associations.

By using the standard for condition assessment, building inspectors can provide property managers with objective data about the condition status of building components. Aggregated condition data could be used for setting condition targets for built assets and for benchmarking. It is anticipated that as a result of the standardisation, condition surveys will become more reliable and as a consequence more popular among large‐scale property owners.

The standard has been introduced recently. At present there is little experience of the use of (aggregated) condition data for maintenance planning and benchmarking built assets.

The standard is a tool to assess the technical status of the properties to underpin the long‐term maintenance expectations. Condition assessment is not meant for preparing the annual maintenance budget and planning of the work. Supplementary information is needed in the phase of preparing for the execution of remedial work.

This paper provides practical tools for condition assessment and maintenance planning.

The research activity presented in this paper has the objective of developing models for the evaluation of technological risk and loss of production due to failures, which are among the criterions that enable the choice of optimal scenarios for energy supply. This activity is based on the European Project “Risk of Energy Availability: Common Corridors for Europe Supply Security” (REACCESS), which aims to develop an analytical tool to analyse scenarios for future secure European Union (EU) energy supply.

The paper proposes an innovative approach, since nowadays a generalised analytic model for risk assessment in large‐scale energetic systems does not exist. In particular, the methodology adopted includes models to assess risk for people safety, risk for the environment and availability for corridors and the related infrastructures. As regards technological risk, accidents producing loss of lives in the population and environmental damage are taken into account; while for the loss of production primary attention is paid to technical failures and maintenance.

Since the analytic models developed perform a large‐scale assessment, they must be flexible and simplified to adapt to different situations and to be easily updated when different future scenarios are investigated. Details of the analysis depend on the precision of data collected and inserted in the models. The damage assessment is affected by deficiency and uncertainties related to territorial and statistical data. Nevertheless, the outcomes obtained for each energy commodity are reasonable and often comparable to literature data.

Based on this study output, technological risk can be considered, more systematically than in the past, in the selection of EU strategies for future energy supply. The corridors social cost is included in future strategies selection, in addition to purely economical and environmental evaluations.

The main purpose of this paper is to examine more closely the tension between, on the one hand, forms of internal school improvement based on internal evaluation measures and, on the other hand, control and legitimisation needs grounded on external evaluation measures.

The clash of these forms of evaluation is at the core of the paper, dealing in particular with the changing evaluation systems in the Cypriot education system. Therefore, the case study approach is utilised here. More specifically, the case of Cyprus is used as a system under transition in order to move from a primarily teacher inspection system, which is externally driven, to a combination of a teacher/school inspection system, which is based on both external and internal processes and is directed at both formative and summative evaluation processes.

It is asserted that the new proposed appraisal system for Cyprus addresses the deficiencies of the current evaluation system and generally aims at achieving a balance between external and internal processes. Finally, the conditions for the acceptance and successful implementation of new evaluation systems are described.

The results of the study constitute a rich setting of future developments for Cyprus schools with regard to important issues such as school accountability, school improvement, teacher evaluation, internal school evaluation and external inspection, through the proposed appraisal system. Therefore, the paper provides an important source of information for those who have the responsibility of creating educational policy and planning for the years to come in the area of teacher appraisal.

The recent global outbreak of Severe Acute Respiratory Syndrome has aroused public concern on environmental health and hygiene. Develops a practical assessment scheme for assessing the health and hygiene performance of apartment buildings in Hong Kong. The scheme involves assessing a hierarchy of building factors that have a bearing on environmental qualities, and thus occupants’ health. Proposes an index method to integrate the assessment outcomes into a simple and user‐ friendly performance indicator for public consumption. The index can inform the public of the health and hygiene risk of different buildings and facilitate building owners, developers, and government bodies to make more informed and socially responsible decisions on environmental health and hygiene improvement. Although the assessment scheme is tailored for the institutional and cultural settings of Hong Kong, the assessment framework for the development of the scheme is also applicable to other cities.

In reflective writing, students are encouraged to examine their own setbacks and progress. With a shortage of guidance in how to provide feedback to students on this type of writing, teachers are often left to figure it out on the job. The central hypothesis in this paper is that the lens of reflective practice can help focus teacher efforts and ultimately improve both feedback and instruction. The purpose of this paper is not to produce a universal prescription for assessing reflective writing but rather a protocol for teacher reflective practice that can apply to challenging grading and feedback-giving situations.

Student assessment is a chance for teachers to learn about their students’ abilities and challenges and to provide feedback for improvement. Assessment and grading sessions can also become opportunities for teachers to examine their own instructional and assessment practices. This self-examination process, a cornerstone of reflective practice (Schön, 1984), is challenging, but it may be especially valuable when guidelines for feedback and assessment are hard to come by. Such may be said to be the case in student-centered learning environments such as school Fablabs and makerspaces, where stated goals commonly include cultivating learner self-regulation and resilience. These hard-to-measure constructs are typically assessed through analysis of student reflective journals. This in-depth case study uses mixed-methods to examine how a semester-long intervention affected the grading, feedback and instructional practices of a teacher in a hands-on design classroom. The intervention involved 10 grade-aloud sessions using a computer-based rubric tool (Gradescope) and a culminating card-sorting task. The lens of reflective practice was applied to understanding the teacher’s development of their own reflective capabilities.

During the intervention, the participating teacher grappled with grading and feedback-giving dilemmas which led to clarifications of assessment objectives; changes to instruction; and improved feedback-giving practices, many of which persisted after the intervention. The teacher perceived the intervention as adding both rigor and productive “soul-searching” to their professional practice. Lasting changes in feedback behaviors included a comprehensive rubric and an increase in the frequency, specificity and depth of feedback given to student written work.

Significant prior efforts have been directed separately at the use of reflective practice for teachers, in general, and on the feedback and grading of student process journals. This work combines these lines of inquiry in the reflective classroom assessment protocol, a novel on-the-job professional development opportunity that fosters reflective practice in times of assessment to improve instructional and feedback practices.

The purpose of this paper is to provide guidance to those contemplating or preparing to administer a large scale information literacy assessment such as the ETS ICT assessment instrument. The case studies and literature review provide real life examples of how to consider implementing the ETS ICT instrument with special attention to issues such as collaboration, timing, marketing, budgeting, and developing a strategy that includes a discussion of how testing results will inform campus information literacy curriculum development and programming.

The planning and implementation by two California State University campuses that administrated beta test versions of the ETS ICT assessment instrument are documented. Background about ICT and guidance for future administrations of large scale assessments on university and college campuses are discussed.

The paper provides background information, techniques and guidance for academic librarians contemplating future administrations and usages of large scale assessments of student information and communication technology skills, like the ETS ICT assessment. Examples of necessary planning stages and collaboration are provided as well as a discussion of the value of large scale assessments for students, campuses and information literacy programs.

This paper offers guidance for academic librarians and libraries interested in assessing their information literacy programs and/or working within their university to conduct a large scale assessment of student ICT literacy skills using the ETS ICT assessment instrument.

The strategies and ideas presented in this paper will help inform other academic libraries and librarians faced with administrating and implementing a large scale assessment instrument such as the ETS ICT instrument.