Search results

The purpose of this study is to compare the evaluation of search result lists and documents, in particular evaluation criteria, elements, association between criteria and elements, pre/post and evaluation activities, and the time spent on evaluation.

The study analyzed the data collected from 31 general users through prequestionnaires, think aloud protocols and logs, and post questionnaires. Types of evaluation criteria, elements, associations between criteria and elements, evaluation activities and their associated pre/post activities, and time were analyzed based on open coding.

The study identifies the similarities and differences of list and document evaluation by analyzing 21 evaluation criteria applied, 13 evaluation elements examined, pre/post and evaluation activities performed and time spent. In addition, the authors also explored the time spent in evaluating lists and documents for different types of tasks.

This study helps researchers understand the nature of list and document evaluation. Additionally, this study connects elements that participants examined to criteria they applied, and further reveals problems associated with the lack of integration between list and document evaluation. The findings of this study suggest more elements, especially at list level, be available to support users applying their evaluation criteria. Integration of list and document evaluation and integration of pre, evaluation and post evaluation activities for the interface design is the absolute solution for effective evaluation.

This study fills a gap in current research in relation to the comparison of list and document evaluation.

The aims of the paper are to: present a theoretical framework for analysing the functioning and consequences of control systems; understand and compare the structure and impact of control systems on two Italian public administrations also in relation to possible situations of crisis and failure; and examine the possible link between the organisational impact of control systems (and their failure if this is the case) and the characteristics of the sectors involved.

The proposed theoretical framework (analysis of the concept, methods, bodies, uses, coherence and impact of control) is applied through a parallel examination of two cases: control systems in Italian universities and ministries over the past 15 years. The objective is to understand why similar rules concerning control systems bring about different consequences and results.

Analysis over a 15 years period shows the effects of two very different strategic approaches and organisational choices in the face of very similar rules. Acceptance or otherwise of control, professional ability and competence, coherence between methods and the object of control and transparency all appear as discriminating factors in the failure or success of control systems. In universities, taking advantage of existing knowledge and professional skills, combining new internationally recognised assessment techniques with academic tradition and culture, focusing methods on specific activities such as teaching and research and arousing attention through transparency of evaluation outputs are the main factors which keep the risk of failure at bay. Evaluation has to cope with the allocation of resource and power issues, but is beginning to take on importance in change management. On the other hand, the case of the ministries shows how the new public management‐type of control exist in a closed cultural context where methods and instruments are taken up without adapting to core processes and where there is secrecy regarding control outputs.

This is an exploratory study that will have to be confirmed by further verification in the future that could include new cases to make the analysis even more exhaustive. The subject of the paper does not lend itself to quantitative studies.

Not to concentrate only on the methodological aspects of control activities, but also focus on all those features of control systems that permit the actual use of output and bring about concrete results in public administrations.

In the past few years, control systems have very often been introduced into the public sector without paying enough attention to how they really work and their true outcome. This paper faces the problem by putting forward a theoretical framework and empirical evidence that may be used by the academic world and public administration.

Inquiry-Based Science Education (IBSE) is considered as a powerful pedagogical practice for renewing science teaching in and out of school. However, despite the spread of IBSE projects, solid evaluation practices have not yet been adequately developed. Practitioners are often uncomfortable with the evaluation task, usually spending little time on it and using poor tools to evaluate IBSE activities. To investigate this issue, we developed an analysis of the European project INQUIRE within the framework of Evaluation Capacity Building theory (ECB). INQUIRE fosters IBSE activities in schools and botanic gardens, promoting training for teachers and educators working with pupils aged between 9 and 14. ECB aims at developing evaluation capabilities in order to conduct rigorous evaluation. Adopting a mixed-methods approach, we systematically analysed evaluations used within courses held in Italy over two years (2011–2013). Study findings suggest that the INQUIRE project has helped practitioners to improve their ability to conduct effective evaluations. However, we also identified two concerns, as IBSE practices tend to focus on cognitive rather than cooperative teaching elements, and on summative rather than formative evaluation methods. The following four recommendations are consequently suggested: promoting dynamic assessment processes, ensuring clearness and flexibility, valuing tacit knowledge, and fostering cooperation.

The purpose of this paper is to examine how the number of activities within a multi-activity experience influences consumer preferences before and after consumption.

The hypotheses are tested using four experiments and a secondary data set from a river cruise firm that includes first-time river cruise purchases by consumers from this firm between January 2011 and December 2015 (n = 337,457).

Consumers prefer experiences with fewer (vs more) activities before consumption – a phenomenon, this paper calls “activity apprehension” – but prefer experiences with more (vs fewer) activities after consumption. A mediation analysis indicates that this phenomenon occurs because the highly perishable nature of activities makes consumers uncertain about their ability to use all the activities within the experience (usage uncertainty).

Evaluations of a multi-activity experience depend on both the number of activities and on whether the consumer is at the pre- or post-consumption stage of the customer journey. As such, firms looking to sell multi-activity experiences should design and promote these experiences in a way that minimizes activity apprehension.

This study is the first to demonstrate that consumer perceptions of an optimal experience depend on both the number of included activities and on the stage of the customer journey (i.e. pre- or post-purchase). It further contributes to the consumer experience literature by examining an unexplored activity characteristic, perishability, in shaping experiential purchase decisions. Finally, it demonstrates a new way in which experiential purchases differ from tangible product purchases.

The pressure on higher education providers (HEPs) and national programme partnerships to evaluate the impact of widening participation (WP) interventions has intensified as a result of wider changes in higher education (HE) policy and regulation, including the imposition of market forces. This chapter describes how policy stakeholder assumptions about how evaluation works and the outcomes it delivers have evolved over the last two decades. It shows how regulatory emphasis has shifted from a focus on monitoring and tracking, through to a call for return-on-investment analysis, before falling back on a pragmatic theory-informed approach. This chapter goes on to locate WP evaluation in the middle of a paradigm war, caught between proponents of a medicalised trial-based conception of evaluation methodology, and a practitioner-led position, which points to the complex contextual character of WP activities. It continues by exploring some of the many practical challenges faced by WP evaluators and argues that these have contributed to the sector’s perceived failure to deliver robust evidence of the impact of fair access activity. This chapter concludes with a look at the expanding market for WP evaluation products and services, which emerged in response to new flows of WP investment created by the 2012 increase in tuition fees.

The purpose of this paper is to suggest indicators for evaluating Iran national nanotechnology plan at the policy and program levels. Governments are required to evaluate their policies to be accountable to their audience and to improve their policies. The evaluation of nanotechnology policies both in academic and governmental institutions has been mainly confined to measuring the research and development outputs, while explicit models are required for evaluating national nanotechnology policies and programs. In other words, although policies are translated into several activities, it does not mean that the evaluation of activities results in holistic policy evaluation.

The present research uses a qualitative method to explore the process of developing a model for assessing nanotechnology policymaking in Iran. Expert opinions have been collected and analyzed by using local and international surveys to validate different elements of the model.

The model includes the evaluation time-frame, criteria and indicators for policy and program levels. Assessment at program and policy levels, as distinct from activity evaluation, measures the ultimate objectives of a policy intervention, based on which the outcome and impact indicators should be developed. This paper argues that the Iran national nanotechnology policy should be evaluated and revised triennially based on the model in which 46 outcome and impact indicators are included under three general criteria of appropriateness, efficiency and effectiveness. A number of these indicators should be evaluated annually or biannually.

The evaluation of nanotechnology plans has been mainly limited to assessing the research and development outputs that are related to the activity level. The main value of this work is developing outcome and impact indicators that are related to policy and program levels.

The urgency to strengthen the effectiveness and efficiency of public administrations has led to the adoption in the public sector – in Italy as well as in other countries – of tools and models inspired by the total quality management (TQM) approach, such as the common assessment framework (CAF). A parallel need was felt within public structures to train people, the “peers”, who aside from being able to implement the self‐evaluation activities of their own administration, also needed to be equipped with the necessary skills to complete external evaluation activities, namely “peer evaluations”, with the ultimate aim of spreading quality management culture and best practices in the field through an approach based on benchmarking. The purpose of this paper is to present the results of a survey designed to determine the training requirements that a “peer” should acquire in order to perform “evaluation” activities.

The paper has strong empirical connotations and is essentially based on, first, a questionnaire given to evaluators/self‐evaluators to identify the problems emerging with the application of the CAF model in administrations that have adopted it. In this sphere a great deal of attention has been paid to – in addition to the preparation of the questionnaire – the delicate task of codifying the answers to open questions proposed to interviewees; and second, the investigation inherent in the current educational choices in Italy in the TQM arena, specifically addressing the public sector.

Apart from the analysis of data obtained from the investigation, presented with descriptive statistics, the paper identifies the training content necessary: first, to place “peers” in a position to be able to autonomously and fully carry out their evaluation work on the basis of the CAF model; and second, to render the evaluation activities systematically comparable among the various administrations.

At the present time the research deals with the Italian context. However, future investigations involving different European countries could be carried out, taking the present results as a starting point for a benchmarking activity.

The paper puts forward the creation of a “network” of actors who manage all the “peer evaluation” activities, from the provision of the training pack and monitoring the “peer” exchange process, up to benchmarking initiatives and the dissemination of best practices.

The paper presents original data and information in order to identify the training needs of individuals that actually use CAF, starting from the assumption that the training of evaluators is the primary condition for promoting the adoption and diffusion of CAF in Italian public administrations and thus maximizing its benefits.

The purpose of this paper is to examine the status quo of communication evaluation and measurement practices in communication departments of companies, non-profits, and other organizations across Europe.

The study argues that the challenge to conduct reliable measurement is threefold: first, communication professionals have to understand and develop skills how to conduct evaluation; second, they have to evaluate whether communication activities have reached those goals in practice; and finally, they have to use those insights to advance and manage their future activities. These aspects are elaborated in the literature review. A quantitative survey of 1,601 professionals from 40 European countries was conducted to research prerequisites, implementation and benefits of communication measurement and compare practices across types of organizations.

Although robust knowledge of empirical research methods and their application for measuring communication effects is indispensable, many practitioners lack the necessary expertise to conduct reliable evaluation and measurement. Communication departments seldom measure communication effects on stakeholders and organizational goals. Many remain focused on media and channels. Last but not least, organizations do not fully exploit the potential of measurement data for strategically planning future communication activities.

The findings highlight the need to reconsider current education and training in communication research methods and their application in corporate practice. Knowledge about conducting applied research is as important as asking meaningful questions and using insights for management decisions in a corporate environment. Evaluation methods are often discussed, but individual skills and the organizational use of insights are important as well. This might be tackled through additional training in social science research techniques, sophisticated valuation methods, and decision making.

The large-scale study shows that communication measurement practices are still in a nascent stage. Joint efforts of academics and professional associations have not really changed the situation until now. The three dimensions used in this research (skills, practices, and utilization) can be used to assess the measurement readiness of individual organizations, to conduct further research in other regions, and to identify future challenges for advancing the field.