Search results

Proposes a continuous participative evaluation process built on the formative evaluation paradigm. The benefits of this approach are that all the primary or core stakeholders, i.e. the users, top management and the technical specialists involved in the task of delivering information systems benefits, participate in the evaluation and the subsequent decision making associated with the project. These stakeholders are consequently involved in moulding and realising an information system which is targeted to meet real business needs rather than just investment and project management criteria. This approach ensures that high quality information systems that deliver direct business benefits with which the user community can identify are implemented. It implies a new focus that encompasses concentrating on and understanding the business issues and how the information system will deliver real value to the organisation. It is the view of the authors that formative evaluation can help to maximise business value from information systems.

This paper presents a set of basic components which constitutes the experimental setting intended for the evaluation of interactive information retrieval (IIR) systems, the aim of which is to facilitate evaluation of IIR systems in a way which is as close as possible to realistic IR processes. The experimental setting consists of three components: (1) the involvement of potential users as test persons; (2) the application of dynamic and individual information needs; and (3) the use of multidimensional and dynamic relevance judgements. Hidden under the information need component is the essential central sub‐component, the simulated work task situation, the tool that triggers the (simulated) dynamic information needs. This paper also reports on the empirical findings of the metaevaluation of the application of this sub‐component, the purpose of which is to discover whether the application of simulated work task situations to future evaluation of IIR systems can be recommended. Investigations are carried out to determine whether any search behavioural differences exist between test persons‘ treatment of their own real information needs versus simulated information needs. The hypothesis is that if no difference exists one can correctly substitute real information needs with simulated information needs through the application of simulated work task situations. The empirical results of the meta‐evaluation provide positive evidence for the application of simulated work task situations to the evaluation of IIR systems. The results also indicate that tailoring work task situations to the group of test persons is important in motivating them. Furthermore, the results of the evaluation show that different versions of semantic openness of the simulated situations make no difference to the test persons’ search treatment.

Evaluating health information systems is an integral part of the life cycle and development of information systems as it can improve the quality of health care. The purpose of this paper is to introduce a bilingual Web-based repository of health-related software products evaluation tools.

The present paper is an applied-developmental study that includes the stages of analysis, design, implementation and evaluation procedures. By searching valid databases as well as holding focus group meetings with a group of experts, the necessary elements for designing a Web-based repository were identified, and also unified modelling language diagrams were designed by using a visual paradigm. The coding(programming) was conducted based on the Gantlet Web Systems Development Framework at the next stage. Finally, after implementing and testing the system, the content was added to the repository, and then the repository was evaluated in terms of usability testing.

The health informatics evaluation tools (HIET) repository provides a functional and selective environment that facilitates the sharing, online storage and retrieval of assessment tools by the scientific community. The HIET repository is easily accessible at www.hiet.ir/ The website is implemented in structured query language (MySQL), personal homepage. Hypertext Preprocessor (PHP) and Linux, Apache, MySQL, and PHP (LEMP) and supports all major browsers.

The HIET repository, as mentioned earlier, serves as an application environment for sharing, storing and online retrieving the assessment tools of health information systems. Therefore, this tool not only facilitates the search, retrieving and study of many evaluation-related papers, which are time-consuming and stressful for researchers and students but can lead to a faster and more scientific evaluation of information systems.

Information and communications technology (ICT) offers enormous opportunities for individuals, businesses and society. The application of ICT is equally important to economic and non-economic activities. Researchers have increasingly focused on the adoption and use of ICT by small and medium enterprises (SMEs) as the economic development of a country is largely dependent on them. Following the success of ICT utilisation in SMEs in developed countries, many developing countries are looking to utilise the potential of the technology to develop SMEs. Past studies have shown that the contribution of ICT to the performance of SMEs is not clear and certain. Thus, it is crucial to determine the effectiveness of ICT in generating firm performance since this has implications for SMEs’ expenditure on the technology. This research examines the diffusion of ICT among SMEs with respect to the typical stages from innovation adoption to post-adoption, by analysing the actual usage of ICT and value creation. The mediating effects of integration and utilisation on SME performance are also studied. Grounded in the innovation diffusion literature, institutional theory and resource-based theory, this study has developed a comprehensive integrated research model focused on the research objectives. Following a positivist research paradigm, this study employs a mixed-method research approach. A preliminary conceptual framework is developed through an extensive literature review and is refined by results from an in-depth field study. During the field study, a total of 11 SME owners or decision-makers were interviewed. The recorded interviews were transcribed and analysed using NVivo 10 to refine the model to develop the research hypotheses. The final research model is composed of 30 first-order and five higher-order constructs which involve both reflective and formative measures. Partial least squares-based structural equation modelling (PLS-SEM) is employed to test the theoretical model with a cross-sectional data set of 282 SMEs in Bangladesh. Survey data were collected using a structured questionnaire issued to SMEs selected by applying a stratified random sampling technique. The structural equation modelling utilises a two-step procedure of data analysis. Prior to estimating the structural model, the measurement model is examined for construct validity of the study variables (i.e. convergent and discriminant validity).

The estimates show cognitive evaluation as an important antecedent for expectation which is shaped primarily by the entrepreneurs’ beliefs (perception) and also influenced by the owners’ innovativeness and culture. Culture further influences expectation. The study finds that facilitating condition, environmental pressure and country readiness are important antecedents of expectation and ICT use. The results also reveal that integration and the degree of ICT utilisation significantly affect SMEs’ performance. Surprisingly, the findings do not reveal any significant impact of ICT usage on performance which apparently suggests the possibility of the ICT productivity paradox. However, the analysis finally proves the non-existence of the paradox by demonstrating the mediating role of ICT integration and degree of utilisation explain the influence of information technology (IT) usage on firm performance which is consistent with the resource-based theory. The results suggest that the use of ICT can enhance SMEs’ performance if the technology is integrated and properly utilised. SME owners or managers, interested stakeholders and policy makers may follow the study’s outcomes and focus on ICT integration and degree of utilisation with a view to attaining superior organisational performance.

This study urges concerned business enterprises and government to look at the environmental and cultural factors with a view to achieving ICT usage success in terms of enhanced firm performance. In particular, improving organisational practices and procedures by eliminating the traditional power distance inside organisations and implementing necessary rules and regulations are important actions for managing environmental and cultural uncertainties. The application of a Bengali user interface may help to ensure the productivity of ICT use by SMEs in Bangladesh. Establishing a favourable national technology infrastructure and legal environment may contribute positively to improving the overall situation. This study also suggests some changes and modifications in the country’s existing policies and strategies. The government and policy makers should undertake mass promotional programs to disseminate information about the various uses of computers and their contribution in developing better organisational performance. Organising specialised training programs for SME capacity building may succeed in attaining the motivation for SMEs to use ICT. Ensuring easy access to the technology by providing loans, grants and subsidies is important. Various stakeholders, partners and related organisations should come forward to support government policies and priorities in order to ensure the productive use of ICT among SMEs which finally will help to foster Bangladesh’s economic development.

It is widely accepted that the use of information and communication technology (ICT) in the healthcare sector offers great potential for improving the quality of services provided, the efficiency and effectiveness of personnel, and also reducing organizational expenses. This paper seeks to examine various hospital information system (HIS) evaluation methods.

In this paper a comprehensive search of the literature concerning the evaluation of complex health information systems is conducted and used to generate a synthesis of the literature around evaluation efforts in this field. Three approaches for evaluating hospital information systems are presented – user satisfaction, usage, and economic evaluation.

The main results are that during the past decade, computers and information systems, as well as their resultant products, have pervaded hospitals worldwide. Unfortunately, methodologies to measure the various impacts of these systems have not evolved at the same pace. To summarize, measurement of users' satisfaction with information systems may be the most effective evaluation method in comparison with the rest of the methods presented.

The methodologies, taxonomies and concepts presented in this paper could benefit researchers and practitioners in the evaluation of HISs.

This review points out the need for more thorough evaluations of HISs that look at a wide range of factors that can affect the relative success or failure of these systems.

This study aimed to analyze existing problems in the dissemination and management of emergency information on social media platforms, improve social media users' experience regarding such information, increase the efficiency of emergency information dissemination and curb the spread of misinformation.

In this study, the emergency information quality on social media platforms was examined. Based on the evaluation principles of the quality of mature information, social media information characteristics and the rules of emergency information dissemination, combined with relevant academic research results, an index to evaluate the quality of emergency information on social media was constructed. In addition, the authors have introduced cloud theory as an information quality evaluation method and used social media users' emotional characteristics to assess information quality evaluation results. A comprehensive system for evaluating emergency information quality, including indexes, methods and detection strategies was established. Based on a comprehensive system, a case study was conducted on the forest fires in Sichuan Province and the African swine fever events as reported on the Zhihu platform. In accordance with the results of the case study, the authors expanded the research and introduced the emotional characteristics of social media users as an independent evaluation dimension to evaluate the quality of emergency information on social media.

The comprehensive system's effectiveness was verified through the case study. Further, it was found that users' emotional characteristics (reflected in their information behavior) are inconsistent with their evaluation of websites' information quality regarding major emergencies. Integrating users' emotional characteristics into the information evaluation system can enhance its effectiveness following major emergencies.

First, an evaluation index system of emergency information quality on social media about major emergencies was offered. Unlike the commonly available index system for information quality evaluation, this proposed evaluation index system not only accounted for the characteristics of social media, such as massive disordered information, multiple information sources and rapid dissemination, but also for the characteristics of emergency events, such as variability and the absence of precursors. This proposed evaluation index system enhances the pertinence of the information quality evaluation and compensates for the shortcoming that the current research only focuses on evaluating social media information quality in a broad context, but pays insufficient attention to major emergencies. Second, cloud theory was introduced as a method to evaluate the emergency information quality found on social media. Existing research has primarily included the use of traditional statistical methods, which cannot transform numerical values into qualitative concepts effectively. Various indeterminate factors inevitably affect the quality of emergency information on social media platforms, and the traditional methods cannot eliminate this uncertainty in the evaluation process. The method to assess emergency information quality based on cloud theory can effectively compensate for the gaps in the research and improve the accuracy of information quality assessment. Third, the inspection and the dynamic adjustment of assessment results are absent in the research on information quality assessment, and the research has relied principally on the information users' evaluation and has paid insufficient attention to their attitudes and behaviors toward information. Therefore, the authors incorporated users' emotional characteristics into the evaluation of emergency information quality on social media and used them to test the evaluation results so that the results of the information quality assessment not only include the users' explicit attitudes but also their implicit attitudes. This enhances the effectiveness of the information quality assessment system. Finally, through this case study, it was found that an inconsistency exists between user evaluation and user emotional characteristics after major emergencies. The reasons for this phenomenon were explained, and the necessity of integrating user emotional characteristics into information quality assessment was demonstrated. Based on this, the users' emotional characteristics were used as a separate evaluation dimension for assessing the quality of emergency information on social media. Compared with assessing the quality of general information, integrating the user's emotional characteristics into the evaluation index system can lead the evaluation results to include not only the users' cognitive evaluation but also their emotional experience, further enhancing their adaptability.

This paper proposes a new hierarchical approach toward MIS evaluation, which emphasizes the evaluation of the strategic aspects of MIS, as well as the evaluations of the activities of MIS functions and the quality of application systems. Based on the previous studies and the preliminary field survey, concepts and contents of three hierarchical evaluation types such as system‐oriented, function‐oriented and strategy‐oriented evaluations were presented, and four propositions were drawn. The surveys of 130 Korean firms, conducted in 1991 and 1997 successively, show that majority of firms use system‐oriented or function‐oriented evaluation type, although the usage rate of strategy‐oriented type is slightly higher in 1997 than in 1991 and most MIS managers consider the strategy‐oriented evaluation as the ideal one. It is also found that the firms of higher MIS maturity use the strategy‐oriented evaluation type, and the firms with strategy‐oriented evaluation type show a higher MIS performance. Results of this study imply that strategic roles of MIS should be more emphasized and different evaluation types should be used along with MIS maturity. In addition, the design of MIS evaluation framework should be done carefully in the strategic and managerial contexts, because MIS performance can vary with evaluation type. Finally, some limitations of the study and directions for further research were suggested.

Model description, observations, and prior knowledge are the three main sources of information that are used for the evaluation and estimation of complex systems. Here, model description is given based on the physical mechanism and dynamics of the system. Observational data represent a very important method of verification and can be applied to validate the model. Prior knowledge can help to provide additional information when observation data are not sufficient. On the basis of clearly carding the relationship between a system's observational quantities and the ultimate indices of the observation, the purpose of this paper is to establish the systemic yoyo model for systems evaluation.

Recent advances in systems science in general and the systemic yoyo model in particular are employed in this research as the fundamental logic of reasoning and thread of thinking.

After analyzing the characteristics and connections between the three main sources of information – model description, prior knowledge, and observational data – used in system evaluation and estimation, the authors derive the conservation law of information for system evaluation and estimation and analyze the transitional direction of the process of system evaluation and estimation.

This work lays down the theoretical basis for why certain procedures widely applied in system evaluations and estimations are correct and sound, on which reliable scientific conclusions can be drawn.

This work is the first of its kind that investigates the systemic foundation underlying the commonly applied procedures of system evaluations and estimations.

Puts forward a framework for carrying out user evaluations of expert systems, which is based on user evaluations carried out in a large British company over a period of three years. The aim of this framework is to ensure that the aims of the expert system designers and the users of the system are congruent. This in turn will make it more likely that the expert system will be a success. The framework covers aspects such as user type, updating knowledge, training, the value of the system and the human/computer interface.