To read this content please select one of the options below:

A rule-based automated machine learning approach in the evaluation of recommender engine

Rajat Kumar Behera (Indian Institute of Management Ranchi, Ranchi, India) (School of Computer Engineering, Kalinga Institute of Industrial Technology, Bhubaneswar, India)
Pradip Kumar Bala (Area of Information Systems and Business Analytics, Indian Institute of Management Ranchi, Ranchi, India)
Rashmi Jain (Montclair State University, Montclair, New Jersey, USA)

Benchmarking: An International Journal

ISSN: 1463-5771

Article publication date: 14 August 2020

Issue publication date: 8 October 2020

382

Abstract

Purpose

Any business that opts to adopt a recommender engine (RE) for various potential benefits must choose from the candidate solutions, by matching to the task of interest and domain. The purpose of this paper is to choose RE that fits best from a set of candidate solutions using rule-based automated machine learning (ML) approach. The objective is to draw trustworthy conclusion, which results in brand building, and establishing a reliable relation with customers and undeniably to grow the business.

Design/methodology/approach

An experimental quantitative research method was conducted in which the ML model was evaluated with diversified performance metrics and five RE algorithms by combining offline evaluation on historical and simulated movie data set, and the online evaluation on business-alike near-real-time data set to uncover the best-fitting RE.

Findings

The rule-based automated evaluation of RE has changed the testing landscape, with the removal of longer duration of manual testing and not being comprehensive. It leads to minimal manual effort with high-quality results and can possibly bring a new revolution in the testing practice to start a service line “Machine Learning Testing as a service” (MLTaaS) and the possibility of integrating with DevOps that can specifically help agile team to ship a fail-safe RE evaluation product targeting SaaS (software as a service) or cloud deployment.

Research limitations/implications

A small data set was considered for A/B phase study and was captured for ten movies from three theaters operating in a single location in India, and simulation phase study was captured for two movies from three theaters operating from the same location in India. The research was limited to Bollywood and Ollywood movies for A/B phase, and Ollywood movies for simulation phase.

Practical implications

The best-fitting RE facilitates the business to make personalized recommendations, long-term customer loyalty forecasting, predicting the company's future performance, introducing customers to new products/services and shaping customer's future preferences and behaviors.

Originality/value

The proposed rule-based ML approach named “2-stage locking evaluation” is self-learned, automated by design and largely produces time-bound conclusive result and improved decision-making process. It is the first of a kind to examine the business domain and task of interest. In each stage of the evaluation, low-performer REs are excluded which leads to time-optimized and cost-optimized solution. Additionally, the combination of offline and online evaluation methods offer benefits, such as improved quality with self-learning algorithm, faster time to decision-making by significantly reducing manual efforts with end-to-end test coverage, cognitive aiding for early feedback and unattended evaluation and traceability by identifying the missing test metrics coverage.

Keywords

Acknowledgements

Funding: The author(s) received no financial support for the publication of this article.Conflict of interest: The authors declare that there is no conflict of interests regarding the publication of this paper.

Citation

Behera, R.K., Bala, P.K. and Jain, R. (2020), "A rule-based automated machine learning approach in the evaluation of recommender engine", Benchmarking: An International Journal, Vol. 27 No. 10, pp. 2721-2757. https://doi.org/10.1108/BIJ-01-2020-0051

Publisher

:

Emerald Publishing Limited

Copyright © 2020, Emerald Publishing Limited

Related articles