Evaluating knowledge management systems efficacy and effectiveness in a design science context

Abstract

Purpose

This paper aims to propose a framework for the evaluation of artefacts in Design Science and test it using an exemplar case of a knowledge management system (KMS) developed for an Australian refrigeration manufacturing company.

Design/methodology/approach

The research uses Design Science research methodology in a specific case study context. The artefact studied was developed using an ontology based on an engineering design conceptualisation and created using an ontology generator, Protégé. Research data for the evaluation of the framework were collected using a combination of document analysis, interviews, shadowing and observations.

Findings

The evaluation framework developed for the research and applied to the KMS specifically built for the company was shown to be useful in determining the efficacy and effectiveness of the research outcomes in terms of usefulness to the company engineers in the technical analysis of their work, and for the CEO and COO as part of their strategic planning for the company. The evaluation framework helped the researcher and the engineers as collaborators to demonstrate the extent of improvement in the design and build processes in the company.

Originality/value

Prior research in both Information System and Design Science has not provided a specific, generalizable, evaluation framework for system developers to use as a guide during the systems development process. This research proposes an evaluation framework which covers all broad aspects of evaluation and efficacy, accepting that evaluation frameworks must be flexible in enabling changes to accommodate variations in the types and purposes of artefacts developed.

Citation

Kanjanabootra, S., Corbitt, B. and Nicholls, M. (2013), "Evaluating knowledge management systems efficacy and effectiveness in a design science context", Journal of Systems and Information Technology, Vol. 15 No. 4, pp. 324-346. https://doi.org/10.1108/JSIT-08-2013-0041