Evaluating knowledge management systems efficacy and effectiveness in a design science context

Sittimont Kanjanabootra (School of Architecture and Built Environment, The University of Newcastle, Newcastle, Australia)
Brian Corbitt (College of Business, RMIT University, Melbourne, Australia)
Miles Nicholls (Graduate School of Business and Law, RMIT University, Melbourne, Australia)

Journal of Systems and Information Technology

ISSN: 1328-7265

Publication date: 11 November 2013



This paper aims to propose a framework for the evaluation of artefacts in Design Science and test it using an exemplar case of a knowledge management system (KMS) developed for an Australian refrigeration manufacturing company.


The research uses Design Science research methodology in a specific case study context. The artefact studied was developed using an ontology based on an engineering design conceptualisation and created using an ontology generator, Protégé. Research data for the evaluation of the framework were collected using a combination of document analysis, interviews, shadowing and observations.


The evaluation framework developed for the research and applied to the KMS specifically built for the company was shown to be useful in determining the efficacy and effectiveness of the research outcomes in terms of usefulness to the company engineers in the technical analysis of their work, and for the CEO and COO as part of their strategic planning for the company. The evaluation framework helped the researcher and the engineers as collaborators to demonstrate the extent of improvement in the design and build processes in the company.


Prior research in both Information System and Design Science has not provided a specific, generalizable, evaluation framework for system developers to use as a guide during the systems development process. This research proposes an evaluation framework which covers all broad aspects of evaluation and efficacy, accepting that evaluation frameworks must be flexible in enabling changes to accommodate variations in the types and purposes of artefacts developed.



Kanjanabootra, S., Corbitt, B. and Nicholls, M. (2013), "Evaluating knowledge management systems efficacy and effectiveness in a design science context", Journal of Systems and Information Technology, Vol. 15 No. 4, pp. 324-346. https://doi.org/10.1108/JSIT-08-2013-0041

Download as .RIS



Emerald Group Publishing Limited

Copyright © 2013, Emerald Group Publishing Limited

Please note you might not have access to this content

You may be able to access this content by login via Shibboleth, Open Athens or with your Emerald account.
If you would like to contact us about accessing this content, click the button and fill out the form.
To rent this content from Deepdyve, please click the button.