Improving the design of a recommendation system using evaluation criteria and metrics as a guide

Abstract

Purpose

The roles recommendation systems play in health care have become crucial in achieving effective care and in meeting the needs of modern care giving. As a result, efforts have been geared toward using recommendation systems in the management of chronic diseases. Effectiveness of these systems is determined by evaluation following implementation and before deployment, using certain metrics and criteria. The purpose of this study is to ascertain whether consideration of criteria during the design of a recommendation system can increase acceptance and usefulness of the recommendation system.

Design/methodology/approach

Using survey-style requirements gathering method, the specific health and technology needs of people living with chronic diseases were gathered. The result was analyzed using quantitative method. Sets of harmonized criteria and metrics were used along with requirements gathered from stakeholders to establish relationship among the criteria and the requirements. A matching matrix was used to isolate requirements for prioritization. These requirements were used in the design of a mobile app.

Findings

Matching criteria against requirements highlights three possible matches, namely, exact, inferential and zero matches. In any of these matches, no requirement was discarded. This allows priority features of the system to be isolated and accorded high priority during the design. This study highlights the possibility of increasing the acceptance rate and usefulness of a recommendation system by using metrics and criteria as a guide during the design process of recommendation systems in health care. This approach was applied in the design of a mobile app called Recommendations Sharing Community for Aged and Chronically Ill People. The result has shown that with this method, it is possible to increase acceptance rate, robustness and usefulness of the product.

Research limitations/implications

Inability to know the evaluation criteria beforehand, inability to do functional analysis of requirements, lack of well-defined requirements and often poor cooperation from people living with chronic diseases during requirements gathering for fear of stigmatization, confidentiality and privacy breaches are possible limitations to this study.

Practical implications

The result has shown that with this method, it is possible to isolate more important features of the system and use them during the design process, thereby speeding up the design and increasing acceptance rate, robustness and usefulness of the system. It also helps to see in advance the likely features of the system that will enhance its usefulness and acceptance, thereby increasing the confidence of the developers in their ability to deliver a system that will meet users’ needs. As a result, developers know beforehand where to concentrate their efforts during system development to ascertain the possibility of increasing usefulness and acceptance rate of a recommendation system. In addition, it will also save time and cost.

Originality/value

This paper demonstrates originality by highlighting and testing the possibility of using evaluation criteria and metrics during the design of a recommender system with a view to increasing acceptance and enhancing usefulness. It also shows the possibility of using the metrics and criteria in system’s development process for an exercise other than evaluation.

Citation

Afolabi, A.O. and Toivanen, P. (2019), "Improving the design of a recommendation system using evaluation criteria and metrics as a guide", Journal of Systems and Information Technology, Vol. 21 No. 3, pp. 304-324. https://doi.org/10.1108/JSIT-01-2019-0019