Search results

A meta-evaluation is an assessment of evaluation practices. Meta-evaluations include assessments of validity and usefulness of two or more studies that focus on the same issues. Every performance audit is grounded explicitly or implicitly in one or more theories of program evaluation. A deep understanding of alternative theories of program evaluation is helpful to gain clarity about sound auditing practices. We present a review of several theories of program evaluation.

This study includes a meta-evaluation of seven government audits on the efficiency and effectiveness of tourism departments and programs. The seven tourism-marketing performance audits are program evaluations for: Missouri, North Carolina, Tennessee, Minnesota, Australia, and two for Hawaii. The majority of these audits are negative performance assessments. Similarly, although these audits are more useful than none at all, the central conclusion of the meta-evaluation is that most of these audit reports are inadequate assessments. These audits are too limited in the issues examined; not sufficiently grounded in relevant evaluation theory and practice; and fail to include recommendations, that if implemented, would result in substantial increases in performance.

This paper seeks to overcome the mismatch between evaluation reports and the expectations of the target audience, by identifying crisis management professionals' expectations.

An adapted stakeholder information analysis was used to survey the expectations of 84 crisis management professionals in the Netherlands. A general inductive analysis was applied to qualitative data, from which five main themes emerged: purpose; object or focus; reasoning and (meta) analysis; result or conclusion, and the overall design of the evaluation.

Currently, evaluation reports are seen merely as a way to share experience and support thinking about how to avoid repeating mistakes. However, most respondents expected them to contribute to learning and support improvement. They should provide actionable feedback on what could be done differently or better, and indicate how this can be achieved. Respondents emphasised the need to focus on the human factor and not neglect the context. The wide variety of views underlined that it is difficult to create one evaluation product that meets all expectations.

Although some major themes clearly emerged from the data, it is unclear how they relate to each other, and their relative importance. In addition, no distinction is made between evaluations of real events and simulations.

Users should be encouraged to provide input into the evaluation process by clarifying their needs and how they use evaluation reports.

This research is the first attempt to identify user expectations regarding what constitutes an effective evaluation.

Changes in the format of library materials, increased amounts of information, and the speed at which information is being produced have created an unrelenting need for training for library staff members. Additionally, library employees are retiring in greater numbers and their accompanying expertise is being lost. The purpose of this study was to document evaluation practices currently used in library training and continuing education programs for library employees, including metrics used in calculating return-on-investment (ROI). This research project asked 272 library training professionals to identify how they evaluate training, what kind of training evaluation practices are in place, how they select programs to evaluate for ROI, and what criteria are important in determining an effective method for calculating ROI.

The study here helps to fill the gap between the current practices of management performance audits for firms and government agencies. The study advances recent theories of program evaluation and marketing management auditing. While the application in this chapter refers to government agencies managing destination marketing programs (tourism agencies), the algorithmic model construction is applicable for all management audits. The study applies the perspectives from two streams of theory to describe five relevant activities for managing destination marketing programs: scanning, planning, implementation, assessing, and administering. The analysis proposes impact assessments to improve management performances of DMOs via checklists for assessing the quality of information in tourism-management performance audits. Checklists can serve as a management tool by management performance auditors and by DMO executives to enhance the quality in executing destination marketing programs. A meta-evaluation of 10 tourism management audit reports identifies good and bad practices. The findings indicate that substantial improvements are possible in the practice of DMO’s management performance auditing, and the proposed checklist may ensure both high quality performance audit reports and improved performances in DMO practices.

This paper aims to address the reporting of validation and evaluation criteria in qualitative management accounting studies, which is a topic of critical debate in qualitative social science research. The objective of this study is to investigate the ways researchers have reported the use of evaluation criteria in qualitative management accounting studies and whether they are associated with certain paradigmatic affiliations.

Building on the work of Eriksson and Kovalainen [Eriksson and Kovalainen (2008) Qualitative Methods in Business Research. London, Sage], the following three approaches are examined: the adoption of classic concepts of validity and reliability, the use of alternative concepts and the abandonment of general evaluation criteria. Content analysis of 212 case and field studies published during 2006 to February 2015 was conducted to be able to offer an analysis of the most recent frontiers of knowledge.

The key empirical results of this study provide partial support for the theoretical expectations. They specify and refine Eriksson and Kovalainen’s (2008) classification system, first, by identifying a new approach to evaluation and validation and, second, by showing mixed results on the paradigmatic consistency in the use of evaluation criteria.

This paper is not necessarily exhaustive or representative of all the evaluation criteria developed; the authors focused on the explicit reporting of criteria only and the findings cannot be generalized. Somewhat different results might have been obtained if other journals, other fields of research or a longer period were considered.

The findings of this study enhance the knowledge of alternative approaches and criteria to validation and evaluation. The findings can aid both in the evaluation of management accounting research and in the selection of appropriate evaluation approaches and criteria.

This paper presents a synthesis of the literature (Table I) and new empirical findings that are potentially useful for both academic scholars and practitioners.

Many development interventions fail to report results that are important to local people (intended beneficiaries of the intervention) but not of strategic importance to the donors funding the work. Failure to report unexpected results, or those not linked to strategic goals, contributes to an overly negative view from external evaluations by donors and agencies. The causes of the mismatch between actual and demonstrated results failure were studied through stakeholder interests. The paper aims to discuss these issues.

Nine project and programme managers of similar but unrelated projects were interviewed. From the interviews, previous studies and project publications, the challenges posed by differing interests and different perceptions in reporting stakeholder activities, outputs and outcomes, were identified. The complex environment of many development interventions was analysed and the work was contextualised with a peacebuilding project in Sri Lanka, which the author has previously studied. A stakeholder role and perception analysis was used to map the challenges at four times in the project cycle, producing a dynamic stakeholder analysis.

The failure to fully report intervention results was linked to the changing role of competing stakeholder interests as a project proceeds, the conflicting perceptions of stakeholders, the structural over-simplification of a complex environment and power differentials that allow donors to misappropriate the role of clients.

Current practice in designing and evaluating projects needs to improve reporting of beneficiary interests.

To the author’s knowledge there are no prior publications in this area of research (under-reporting of development intervention results); the paper is considered highly original.

The purpose of this paper is to reflect on the reporting of cybercrime in Australia, specifically the reporting of fraud.

Using an evaluation of the (former) Australian Cybercrime Online Reporting Network (ACORN), this paper provides a commentary on the report findings, including both positive and negative aspects of the reporting mechanism.

The paper focuses on three key challenges that were identified in the evaluation: victim satisfaction; quality of data; and unintended consequences. Each of these topics is outlined and located within a broader context to better understand the evaluation findings.

This viewpoint paper is a commentary based upon an evaluation of ACORN which itself has several limitations of its methodology. Despite this, the evaluation provides important insights into the challenges that currently exist with the online reporting of fraud offences both in Australia, and worldwide.

This paper emphasises the structural and systemic issues that exist for the reporting of fraud in Australia. It therefore warns against placing blame exclusively on police agencies as responsible for these results. Instead, it advocates the need for society to take a more coordinated and collaborative approach to the policing of fraud, which includes law enforcement, government and industry partners.

The paper documents some of the underlying reasons for additional trauma and harm experienced by victims of fraud in their efforts to report their incident and receive what they perceive to be an appropriate criminal justice response. These must be acknowledged in order to make the required change.

This paper is a commentary and reflection on the current way in which fraud is reported in Australia. It points to a need to rethink this approach in some key areas. It highlights the critical need for an education campaign to dispel some of the myths that exist in relation to realistic police responses to fraud, and also calls for the need to consider alternatives to the exclusively online system currently in operation, as well as larger questions about notions of justice and appropriate responses to fraud victims.

This innovative analysis aims to quantify the use of evaluation criteria in telemedicine and to identify current trends in metric adoption. The focus is to determine the frequency of actual performance metric reporting in telemedicine evaluation, in contrast to systematic reviews where assessment of study quality is the goal.

Automated literature search identified telemedicine studies reporting quantitative performance metrics. Studies were classified by telemedicine class; store‐and‐forward (SAF), real‐time consultation (RTC) and telecare (TC), and study stage. Studies were scanned for evaluation metric reporting, i.e. clinical outcomes, satisfaction, patient quality and cost measures.

Evaluation metric use was compared among telemedicine classes, and between pilot and routine use stages. Diagnostic accuracy was reported significantly more frequently in pilots for RTC and TC. Cost measures were more frequently reported in routine use for TC. Clinical effectiveness and hospital attendance were better reported in routine use for SAF. Comparison also revealed different evaluation strategies. In pilots, SAF favoured diagnostic accuracy, compared to RTC and TC. TC preferred clinical effectiveness evaluations and TC more frequently assessed patient satisfaction. Cost was only reported in less than 20 per cent of studies, but most frequently in RTC. Routine use led to increased reporting of all metrics, except diagnostic accuracy. Clinical effectiveness reporting increased significantly with routine use for RTC and SAF, but declined for TC.

Clinical outcomes and patient satisfaction were reported frequently in telemedicine studies, but reporting of other performance metrics was rare. Understanding current trends in metric reporting will facilitate better design of future telemedicine evaluations.

This paper aims to outline a conceptual framework for the quality evaluation of web-based sustainability reports (SRs) aiding managers in determining and evaluating quality criteria for the sustainability report of their company.

The paper reviews quality criteria in existing standards and guidelines on sustainability reporting and identifies research gaps. A conceptual framework including a multi-method approach for the quality evaluation of SRs is developed and evaluated.

Existing standards and guidelines on sustainability reporting mainly focus on the content of the reports and neglect common information systems (IS) acceptance criteria such as ease of use and visual appeal. The proposed framework directly involves different stakeholder groups and research methodologies into the quality evaluation process.

The limitations of the research approach offer a number of starting points for future research. The proposed framework needs to be further evaluated by a larger number of test users in a more natural use setting.

The application of a multi-method approach as well as the direct involvement of the stakeholders allows for an in-depth quality evaluation of SRs, enabling reporting companies to meet the readers' demand for information on economic, environmental and social activities of the reporting company. Common acceptance factors from the field of IS should be integrated into existing standards and guidelines on sustainability reporting. Coaching of the users through help functions, wizards, instructional videos or avatars is desirable.

The proposed framework applies innovative technologies such as eye-tracking and software-supported attention analysis. By applying the framework to a set of sample reports, its usefulness and applicability are demonstrated.

Global health challenges and threats could be confronted by collaborative efforts of international community. Governance for global health is a set of formal and informal processes, operating beyond state boundaries, and refers to institutions and mechanisms established at the national, regional and international levels. Nordic countries demonstrated a long-standing commitment to development assistance for health (DAH), and more recently to governance for global health. Governance for global health tools could be used effectively to achieve collective solutions for the maintenance and promotion of health as a common good, could ensure accountability and transparency, and reconcile the interests of different actors on the international and national levels. The aim of this paper is to provide an overview of tools and approaches in support of eight sub-functions of governance for global health applied by the Nordic countries. This will help international audience to compare those mechanisms with similar mechanisms that are available or planned in their countries and regions, and may benefit policy scholars and practitioners.

This study uses qualitative review of research literature, policy documents and information available from institutional websites related to the governance of global health in four Nordic countries. In total, 50 selected publications were analyzed using framework synthesis, mapping all findings to 8 dimensions (sub-functions) of governance for global health and related tools.

Review reveals which tools are available, how they have been applied by the Nordic countries and influenced all domains (sub-functions) of governance for global health at different levels: national governments, agencies and networks; bilateral and multilateral partnerships; inter-governmental institutions and international health-related organizations. Common trends and approaches in governance for global health have been formulated.

This study is unique in relation to the prior literature as it looks at the role of Nordic countries in the governance for global health system through the lens of tools applied in support to its sub-functions.