Enhancing quality 4.0 and reducing costs in lot-release process with machine learning-based complaint prediction

Abstract

Purpose

This study proposes a machine learning framework to predict customer complaints from production line tests in an automotive company's lot-release process, enhancing Quality 4.0. It aims to design and implement the framework, compare different machine learning (ML) models and evaluate a non-sampling threshold-moving approach for adjusting prediction capabilities based on product requirements.

Design/methodology/approach

This study applies the Cross-Industry Standard Process for Data Mining (CRISP-DM) and four ML models to predict customer complaints from automotive production tests. It employs cost-sensitive and threshold-moving techniques to address data imbalance, with the F1-Score and Matthews correlation coefficient assessing model performance.

Findings

The framework effectively predicts customer complaint-related tests. XGBoost outperformed the other models with an F1-Score of 72.4% and a Matthews correlation coefficient of 75%. It improves the lot-release process and cost efficiency over heuristic methods.

Practical implications

The framework has been tested on real-world data and shows promising results in improving lot-release decisions and reducing complaints and costs. It enables companies to adjust predictive models by changing only the threshold, eliminating the need for retraining.

Originality/value

To the best of our knowledge, there is limited literature on using ML to predict customer complaints for the lot-release process in an automotive company. Our proposed framework integrates ML with a non-sampling approach, demonstrating its effectiveness in predicting complaints and reducing costs, fostering Quality 4.0.

Citation

Lobo, A., Sampaio, P. and Novais, P. (2024), "Enhancing quality 4.0 and reducing costs in lot-release process with machine learning-based complaint prediction", The TQM Journal, Vol. 36 No. 9, pp. 175-192. https://doi.org/10.1108/TQM-10-2023-0344

1. Introduction

Industry 4.0 (I4.0) companies rely on digitisation, automation and real-time operations to improve customer service (Rojko, 2017). This movement is driven by big data and artificial intelligence (AI) tools, which are the core components of improving manufacturing processes and service quality (Escobar et al., 2021). Quality 4.0 (Q4.0) addresses this challenge and can be seen as a digital transformation strategy in which quality and performance goals are the top priority (Radziwill, 2020).

The lot-release decision process in industrial operations can significantly impact efficiency and service quality. Typically governed by heuristic rules, integrating machine learning (ML) can optimise it, enhancing lot-release decision quality and customer satisfaction by minimising complaints. To do this, managing imbalanced data from diverse sources is crucial for effective implementation (Fathy et al., 2021). This is the case for the studied company's software application, which currently relies on heuristic rules and must deal with heavily imbalanced datasets.

Several studies have proposed frameworks and models to improve quality control in manufacturing. Villanueva Zacarias et al. (2018) introduce a framework for selecting and configuring ML-based data analytics solutions, considering factors such as data quality and algorithm selection. Cho et al. (2022) focus on data preprocessing, using different methods to address missing values and data imbalance. This paper proposes a framework that integrates an ML model to improve the lot-release decision process, reduce quality costs and contribute to the adoption of Q4.0. This approach focusses on the last stage of the production line, gathering information from automatic production tests and repairs generated along the different production stages. Based on this information, four ML algorithms XGBoost (XGB), LightGBM (LGBM), CatBoost (CatB) and Random Forest (RF)) were conceived, tuned, evaluated and compared to classify the occurrence of a customer complaint. Two non-sampling approaches (cost-sensitive learning and threshold moving) were considered to deal with imbalanced data.

The remainder of this paper is organised as follows. Section 2 presents the related work and background. Section 3 describes the methods used to deal with imbalanced datasets, the ML algorithms considered, the evaluation metrics and how the data were collected and preprocessed. Section 4 describes the experiments carried out, and Section 5 discusses the results obtained. Finally, in Section 6 are given the main conclusions and future work directions.

2. Literature review

Improving the quality of products and services is an essential component of competitiveness for every company. I4.0 promotes the digitisation of processes to create autonomous systems and integration throughout the supply chain. This environment poses new challenges that are addressed by Q4.0. The implementation of Q4.0 can be seen as a digital transformation strategy where quality and performance are crucial (Radziwill, 2020). In this sense, customer satisfaction and product-related complaints are two of the four vital quality objectives that must be improved with Q4.0 (Dror, 2022). Compared to traditional quality, regarding quality and performance goals, Q4.0 focusses on minimising or eliminating appraisal costs by detecting problems before they occur, which has a positive impact on quality costs (Radziwill, 2020).

2.1 Related work

Over the years, the prediction of anomalies in different stages of production lines using different ML techniques has been studied thoroughly; however, this is not the case for the prediction of customer complaints (Abdelrahman and Keikhosrokiani, 2020). There is little literature on the use of ML techniques to predict them based on the results of tests carried out along production lines.

Chen and Lin (2020) address this topic in a textile company, creating a system to predict the probability of complaints about a new production order based on its inherent characteristics. As customer complaints are relatively rare, they must deal with imbalanced datasets to train the classifiers; to deal with this, they propose an upsampling approach. To evaluate the results of the three ML classifiers tested (Decision Trees, RF and XGB), they use balanced accuracy, which is the arithmetic mean of sensitivity and specificity. In their pipeline, they also consider grid search to find the best hyperparameters. The results show that when upsampling, the grid search, the area under the roc curve (AUC) metric, and the XGB classifier were coupled, the balanced accuracy during validation was maximised and the gap between balanced accuracies during training and validation was minimised.

Yorulmuş et al. (2022) use quality data from a brake assembly line of an automobile manufacturer to develop a predictive quality model to recognise products that passed quality inspection operations without defects but are problematic. To achieve this, they considered several ML algorithms and chose the specificity and negative prediction values to compare them. The values obtained show that the gradient boost and CatB algorithms achieved the best results in detecting rare events. However, despite analysing the classification results of rare events that appear in imbalanced datasets, it is not clear what approach was used to deal with them.

The viability of using ML techniques to predict compliance quality from data of multiple processes was confirmed by Sankhye and Hu (2020). In their study, they focus on analysing data from a large-scale appliance manufacturing plant to design ML-based classification methods to predict manufacturing compliance quality according to the results of quality inspections. They compare RF and XGB, with Cohen’s Kappa as the reference metric and use synthetic minority oversampling technique (SMOTE) to implement an oversampling approach to deal with the imbalanced dataset. They also analysed the impact of the feature engineering process and concluded that the results improved when certain features were created by applying prior domain knowledge to the datasets nature. Overall, the best results were obtained with XGB.

Product quality is vital for customer satisfaction and cost reduction in the automotive industry. Current research primarily targets anomaly prediction at specific production stages, overlooking predicting customer complaints during lot release, which can impact reputation and profitability. This paper addresses this gap by proposing an ML framework that uses production line data to predict customer complaints in the lot-release process, thereby promoting Q4.0 adoption and reducing quality costs. The framework incorporates algorithmic-level methods like cost-sensitive learning and threshold moving to handle imbalanced data effectively. Threshold-moving enables efficient model adjustments without retraining, offering practical and cost-effective solutions. The framework also leverages state-of-the-art tools for automatic feature creation and hyperparameter optimisation, enhancing the predictive model’s robustness and performance.

2.2 Quality 4.0

The concept of Q4.0 has been a subject of considerable debate and discussion amongst researchers and practitioners (Oliveira et al., 2024). Q4.0 incorporates advanced technologies to improve the quality of manufacturing and services, in the context of increasing digitisation of industries (Javaid et al., 2021). It combines quality management with I4.0 to improve organisational performance, innovation and new business models (Antony et al., 2021). The critical success factors (CSFs) associated with the implementation of Q4.0 are in line with I4.0. These include investing in technology, developing the right skills, providing adequate training and knowledge, addressing cybersecurity concerns, having management support, fostering a supportive organisational culture and effectively managing resistance to change ( Antony et al., 2023). This is reflected in the eight identified key ingredients for an effective implementation of Q4.0: handling big data, improving prescriptive analytics, using Q4.0 for effective vertical, horizontal and end-to-end integration, using Q4.0 for strategic advantage, leadership in Q4.0, training in Q4.0, organisational culture for Q4.0 and, top management support for Q4.0 (Sony et al., 2020). These key ingredients play a crucial role in the ability of companies to embrace Q4.0 and are aligned with the five readiness factors for Q4.0 identified by Zulfiqar et al. (2023): top management commitment and support, leadership, organisational culture, employee competency and presence of an ISO Quality Management System (QMS) standard. Integration of I4.0 technologies and digitisation of quality management have a substantial effect on quality technology, processes and people (LNS Research, 2017). LNS Research proposes a framework with 11 axes for Q4.0 that outlines how it can enhance existing capabilities and initiatives whilst providing a perspective on traditional quality methods. Included in these 11 dimensions are analytics and data.

The evolution of quality management has progressed from inspection to total quality management (TQM), with tools aimed at enhancing industrial processes and services (Broday, 2022). This evolution has culminated in the emergence of Q4.0, which builds upon TQM by integrating Big Data and AI (LNS Research, 2017; Escobar et al., 2021). Q4.0 represents a digital transformation strategy focussed on leveraging digital tools to consistently deliver high-quality products. These tools encompass AI, Big Data, Blockchain, Deep Learning, ML, Statistics and Data Science, alongside enabling technologies such as Internet of Things (IoT), Virtual and Augmented Reality, Data Streaming, Sensors and 5G (Radziwill, 2018). Considering the evolution of quality approaches, these tools are instrumental in shaping Q4.0 as a discovery approach:

1. Inspection: Quality assurance was based on inspection, with the use of Walter A. Shewhart’s statistical process control methods.

2. Design: Integration of quality into operations to proactively prevent quality issues, based on W. Edwards Deming’s suggestions.

3. Empowerment: Use of TQM and Six Sigma, where quality is a shared responsibility and people are empowered to participate in continuous improvement.

4. Discovery: In an adaptive environment, quality relies on the quick identification of new data sources, root cause analysis and discovery of new knowledge (Radziwill, 2018).

It is clear that Q4.0 involves a shift from traditional quality methods to a more data-driven approach (Grandinetti et al., 2020; Carvalho et al., 2021). This includes the use of data analytics and ML to identify patterns and trends in quality data, which can be used to improve processes and make better decisions. This also means that Q4.0 places greater emphasis on data collection and analysis, as well as the use of digital tools to manage and track quality metrics (Thekkoote, 2022).

Amongst the technological advancements driving industrial transformation, business analytics stands out as a pivotal enabler of I4.0, playing a crucial role in this process. It is rooted in theoretical concepts like absorptive capacity, dynamic capabilities and data-driven decision-making (Duan et al., 2020). Almazmomi et al. (2021), underscore its significance in fostering a competitive advantage, particularly through nurturing a data-driven culture and enhancing product development within I4.0. Business analytics, by providing data intelligence and expert system components, is instrumental in facilitating the successful implementation of Q4.0 within the broader context of I4.0 (Silva et al., 2021). It extracts meaningful insights from vast industrial data, contributing to digital market transformation (Duan et al., 2021). It is also a key element for Q4.0 alongside data, connectivity and leadership (Thekkoote, 2022). The integration of advanced analytics and big data necessitates the development of an I4.0 analytics platform, transcending mere tools and technology (Gröger, 2018). This aligns with the view that business analytics serves as a strategic resource for gaining a competitive edge in the industrial sector during the I4.0 era. Additionally, Ehret and Wirtz (2017) highlight how the Industrial Internet of Things (IIoT) drives new business models and services, emphasising analytics, including big data and AI as enablers of innovative information and analytical services. Additionally, Fernando et al. (2018) underscore practical big data analytics for predicting market preferences from diverse data sources, particularly in enhancing supply chain performance, an essential aspect of industrial transformation in the I4.0 era.

Q4.0 implementation enhances customer satisfaction, product quality, service quality and competitive advantage (Antony et al., 2023). I4.0 objectives mirror those of the early to mid-1990s but with two shifts: data volume surge and accelerated achievement of quality goals through emerging technologies (Radziwill, 2020). Due to rising customer expectations in the I4.0 and Q4.0 landscape challenge, companies deliver high-quality products at competitive prices (Keller et al., 2014). This underscores the relevance of understanding and managing quality costs.

Schiffauerova and Thomson (2006) analysed the different models that have been proposed to quantify the Cost of Quality (CoQ), each with its unique cost or activity categories. PAF model categorises costs into Prevention, Appraisal and Failure. Crosby’s model splits quality costs into conformant and non-conformant. Opportunity or Intangible cost models extend the PAF model to include opportunity costs. The Process Cost model provides a systematic approach to identifying and analysing process-related costs. Activity-Based Costing (ABC) models categorise costs into value-added and non-value-added activities. I4.0’s digital technologies and data analysis drive a transformation impacting quality costs and quality management. These technologies improve customer satisfaction and reduce quality costs (Saihi et al., 2021). Antony et al. (2023), Sony et al. (2020) and Zulfiqar et al. (2023) highlight technology investment and employee skills as key to Q4.0. Maganga and Taifa (2023), reinforce this, identifying investment in Big Data handling, enabling technologies and human resources skills as main enablers. These costs could be offset by reduced failure costs and increased customer satisfaction, leading to greater market share (Margarida Dias et al., 2021). Tools like IoT, Cyber-Physical Systems (CPS), big data and AI contribute to predictive maintenance implementation, reducing costs and preventing failures (Lee et al., 2019). Q4.0 provides a management framework based on increasing customer loyalty and decreasing costs (Javaid et al., 2021).

Figure 1 represents the evolution of quality costs until the Q4.0 objectives were fully achieved (right column) (DeFeo, 2018). As shown, appraisal costs are minimised or eliminated, as well as internal and external failures. In general, Q4.0 can be seen as a holistic approach to quality that uses advanced technologies to improve efficiency, reduce costs and enhance customer satisfaction.

Advancements in quality management, though significant, bring new challenges. Digitisation of manufacturing processes alters communication, consumption patterns and value creation, influencing market strategies and product life cycles (Paritala et al., 2017). Organisations must comprehend and adapt to these changes, analysing their impact on quality management (Corti et al., 2021; Antony et al., 2023). This ensures effective utilisation of Q4.0 to enhance customer satisfaction, enterprise efficiency and competitiveness (Liu and Gu, 2023). Q4.0, which is related to the digitalisation of quality work in the context of I4.0, is a relatively new and evolving area. Corti et al. (2021) and Ranjith Kumar et al. (2021) highlight this transition challenge and opportunities, with Corti providing a comprehensive framework for Q4.0 adoption and Kumar proposing a conceptual framework for quality in the digital transformation context. Margarida Dias et al. (2021) stress the need for a unified Q4.0 definition. Sureshchandar (2022) underscores the critical role of traditional quality elements such as leadership, customer focus and data-driven decision-making in the digital transformation journey, which aligns with the 11 axes of Q4.0 proposed by LNS Research (2017). These authors collectively suggest that whilst digitalisation presents new challenges, it also offers significant potential for enhancing quality management.

3. Materials and methods

This section details the materials and methods used in this study. The methodology, proposed framework, data preparation, data exploration, ML models used and evaluation metrics are described below.

3.1 Lot-release challenge

An automotive company is committed to minimising customer complaints, reducing costs and improving overall quality on its way to Q4.0. During different stages of its production line, different tests are performed. The products are packed in pallets that the company defines as a lot. The decision to release the lot is made in the last stage before shipping the products. This can have a great impact on customer satisfaction. Currently, the company uses a software application that manages this process by applying a set of rules that were defined heuristically. The lot is locked by default, and it is the software system that decides to release them by applying a set of rules that were manually defined by quality engineers based on their perception and knowledge. The actual rules are related to the following.

1. Total repairs by part number, station and type of defect

2. Faults detected in critical stations

3. Part number blocked by quality team

After unlocking the lot, risk analysis is performed to decide whether it can be sent to customers.

Taking into account this manual system and given the vast amount of data that needs to be processed, an efficient method of handling it is crucial. Another factor to consider is the promotion of Q4.0 adoption in an effective manner. With these considerations in mind, the proposed framework incorporates an ML model to address these problems (Sarker, 2021). This will allow companies to improve their process, reduce costs and easily adjust their predictive models efficiently. Figure 2 reflects this by adding an ML module that collects information from tests and repairs at different stages to predict the probability of customer complaints before the lot is shipped to customers. In this phase, it is proposed to maintain existing manual rules to promote better decisions. The framework has been tested with real-world data and has shown promising results in terms of its ability to predict customer complaints and its efficiency. Based on the ML results and with continuous training, a more automated process can be evaluated that relies only on the ML model.

The development and evaluation of this ML module followed the Cross-Industry Standard Process for Data Mining (CRISP-DM) methodology (Wirth and Hipp, 2000). This is widely recognised as the standard for implementing data mining projects (Schröer et al., 2021). This structured approach breaks down the life cycle of a data mining project into six phases:

1. Business Understanding: Define project objectives and requirements from a business perspective and formulate a data mining problem definition.

2. Data Understanding: Collect and familiarise with data, identifying quality issues and forming hypotheses for hidden information.

3. Data Preparation: Construct the final dataset from raw data, including selection, transformation and cleaning tasks.

4. Modelling: Select and apply modelling techniques, calibrating parameters for optimal performance. Certain techniques have specific data format requirements, which may require revisiting the data preparation phase.

5. Evaluation: Assess model quality and effectiveness, ensuring alignment with business goals before deployment.

6. Deployment: Structure and present knowledge for customer utilisation, varying in complexity from report generation to recurring data mining procedures.

Figure 3 illustrates these phases and the approach adopted in the studied company, where management and team commitment played a crucial role in navigating each phase.

3.2 Data exploration

Production tests and repair databases of an automotive company were used as data sources for this study, considering the period between January 2019 and April 2020. Millions of automatic tests are produced every day; to handle them, the company stores the data in a Hadoop cluster. The initial dataset has 2,076,529 records and 40 features, 31 related to production tests and 9 related to repairs. Table 1 lists some of them:

3.3 Data preparation

The following constraints were implemented to ensure a representative dataset that includes tests related to complaints and tests without complaints in the period analysed.

1. Select the top 10 products with more complaints.

2. Select a subset of 2 million tests without complaints.

To improve the performance of the ML models, feature engineering was carried out based on the insights gained from the data exploration. Throughout this process, some features were manually created, such as hascomplaint, classified as the target to determine whether a test is related to a complaint or not, hasrepair, which identifies whether a test has a repair or not and srepair, which identifies a specific type of repaired flaw. Most of the new features were created automatically using Featuretools, which implements the concept of deep feature synthesis (Kanter and Veeramachaneni, 2015).

In the data cleaning process, the rows with missing values were dropped. Spearman’s rank correlation coefficient was used to evaluate correlation amongst features, excluding the target and dropping highly correlated ones to avoid multicollinearity.

After this operation, the final dataset comprises 1,552,324 records and 44 features, including 15 new aggregation and transformation features that were automatically created. The distribution of the two classes is as follows: 1.7% come from the class of tests related to customer complaints, and 98.3% come from the other class.

3.4 Evaluation metrics

Due to the need to deal with imbalanced data when predicting customer complaints based on repair and production tests, classification is assessed with a confusion matrix (He and Garcia, 2009). Tailored performance metrics are essential for effectively addressing imbalanced datasets. Precision, Recall, F1-Score and Matthews Correlation Coefficient (MCC) are recognised as suitable measures for evaluating model performance on such datasets (Chicco and Jurman, 2020; Bhadani et al., 2023). These metrics are calculated according to the four categories of confusion metrics: True Positives (TP), True Negatives (TN), False Positives (FP) and False Negatives (FN).

Precision is the fraction of relevant results.

(1)Precision=True Positive(True Positive+False Positive)

Recall is the fraction of positive labels correctly identified by the model.

(2)Recall=True Positive(True Positive+False Negative)

The F1-Score is the harmonic mean of precision and recall. It provides the model’s ability in identifying both classes.

(3)F1−Score=2*Precision*RecallPrecision+Recall

The MCC assesses the correlation between predicted and observed binary classes in classification, yielding a high score when both positive and negative predictions are accurate.

(4)MCC=TP*TN−FP*FN(TP+FP)*(TP+FN)*(TN+FP)*(TN+FN)

3.5 Machine learning algorithms selected

Given the problem’s imbalanced nature, three gradient-boosting (XGB, LGBM, CatB) and a bagging (RF) algorithm were studied. These algorithms are effective in handling imbalanced datasets (Shumaly et al., 2020). Specifically, XGB, LGBM and CatB are efficient, accurate and have a large set of hyperparameters that can be tuned (Bentéjac et al., 2021). All selected algorithms have shown good results using a non-sampling approach to deal with imbalanced datasets (Johnson and Khoshgoftaar, 2022).

3.5.1 Gradient boosting

Boosting is an ensemble learning technique that consists of sequentially applying a number of weak learners (models that perform marginally better than random guessing) to create a strong learner (Sagi and Rokach, 2018). A boosting algorithm assigns varying weights to the output of its estimator and optimises a loss function. In gradient boosting, which combines the gradient descent algorithm and the boosting method, predictors are created consecutively rather than independently, where each tree corrects the error caused by the prior tree (Daoud, 2019).

XGB, LGBM and CatB are all advanced gradient-boosting algorithm implementations. XGB employs innovative regularisation techniques for controlling overfitting, enhancing scalability and speed whilst conserving computational resources (Chen and Guestrin, 2016). LGBM introduces Gradient-based One Side Sampling and Exclusive Feature Bundling techniques, enhancing efficiency and suitability for large-scale data processing (Ke et al., 2017). CatB introduces an innovative approach for handling categorical features and implementation of ordered boosting, improving its effectiveness and performance (Prokhorenkova et al., 2018).

3.5.2 Bagging

Bagging or bootstrap aggregation is an ensemble learning method. In bagging, some samples from the original dataset are randomly generated by replacement, which means that each row can be selected more than once (Breiman, 2001). After multiple data samples are generated, the outcomes generated by these weak models, which are trained independently, are evaluated, and based on the type of task (regression or classification), the average or majority of those predictions result in a more accurate estimate. RF is an extended implementation of bagging. Unlike bagging, where all features are taken into account when splitting a node, in RF, only a subset of the total features is randomly chosen, and the best split feature of the subset is used to split each node in a tree. It is robust against noisy data and outliers, making it less prone to overfitting (Breiman, 2001).

3.6 Cost-sensitive learning and thresholding

In addressing skewed class distributions, two primary strategies are employed: data-level and algorithm-level methods. The latter adjusts the learning algorithm to better manage imbalanced data, incorporating techniques like cost-sensitive learning and thresholding (Haixiang et al., 2017).

Cost-sensitive learning employs a cost matrix where the penalties for misclassification are strategically set higher than those for correct classification. This can be operationalised by adjusting the “class weight” parameter in the selected algorithms to reflect the varying importance of each class. Thresholding involves establishing a decision boundary to predict class membership. For imbalanced datasets, the default threshold often proves inadequate, potentially skewing results. To counter this, the threshold is fine-tuned during training to enhance model performance. To perform this threshold analysis, the Precision-Recall Curve, which illustrates the trade-off between precision and recall across different thresholds, was considered (Davis and Goadrich, 2006; Lobo et al., 2023).

4. Experiments

To conduct the study, four ML models were conceived, tuned and evaluated. The dataset was randomly split using the common ratio of 80% for training, 10% for validation and 10% for testing (Alazba et al., 2023). Finding optimal hyperparameters is crucial for ML algorithm performance, but manual search can be time-consuming. Whilst Grid Search and Random Search are common methods, Bayesian optimisation with the Hyperopt library is more efficient and effective in terms of both accuracy and time (Putatunda and Rama, 2018). Each algorithm has specific hyperparameters (Chen and Guestrin, 2016; Ke et al., 2017; Pedregosa et al., 2011; Prokhorenkova et al., 2018). To improve the F1-Score and control overfitting the following parameters were considered.

1. “n_estimators”: Specifies the number of trees or boosting stages. More trees can improve learning ability but may lead to overfitting.

2. “max_depth”: Controls overfitting by specifying the maximum depth of a tree.

3. “learning_rate”: Controls the weighting of new trees. Lower rates may improve performance but extend training time.

4. “colsample_bytree”: Controls column subsampling to prevent overfitting.

5. “min_samples_leaf”: Specifies the minimum number of samples required at a leaf node, affecting noise capture.

6. “min_samples_split”: Defines the minimum number of samples required for node splitting to control overfitting.

To tune and find the best hyperparameters for these models, cross-validation was used together with Hyperopt. Table 2 depicts the hyperparameter searching space.

This study was implemented using Python, version 3.7, including some libraries such as Pandas, Numpy, MatPlotlib, Hyperopt and Featuretools. All ML models were conceived using Scikit-Learn.

5. Results and discussion

Based on the best hyperparameters found for each model, which are shown in Table 3, four ML models were analysed.

Cost-sensitive learning and thresholding methods were used to deal with an imbalanced dataset of production and repair tests to predict customer complaints. The results obtained for each model based on this approach are presented in Table 4. To evaluate the results, the F1-Score and MCC were the main metrics considered.

Analysing the results, CatB had the best Recall at 66.9% and FN count, whilst XGB excelled in other metrics, including a 72.4% F1-Score and 75.0% MCC. These results were achieved through cost-sensitive learning, adjusting the class weight parameter and threshold moving. The Precision-Recall curve was used to find the optimal threshold for the best F1-Score (Brownlee, 2020). Optimising this value results in the ideal balance between precision and recall. Figure 4 illustrates this process for the XGB model, showing the trade-off between increasing Precision and decreasing Recall. The F1-Score identifies the threshold that achieves the best balance between precision and recall.

Despite limited research on predicting customer complaints from production and repair data for industrial processes, this study leverages insights from related fields. Prior studies underscored the role of ML and AI in the automotive industry (Fernández-López et al., 2022), the significance of ML in predicting and analysing customer complaints (Alarifi et al., 2023) and improving quality prediction (Jung et al., 2021). The obtained results of CatB and XGB align with previous research, demonstrating their efficacy in tackling similar problems (Chen and Lin, 2020; Yorulmuş et al., 2022). Furthermore, we found that LGBM and RF yielded lower F1-Score and MCC values compared to CatB and XGB. Our findings also demonstrate the potential of cost-sensitive learning and thresholding methods to enhance the performance of these models (Petrides and Verbeke, 2022; Coussement, 2014).

The results show that the system can predict a significant number of potential complaints based on production and repair tests. The F1-Score and MCC values indicate a balanced precision and recall, reflecting good-quality classification. In the automotive industry, recall values help reduce 0 km defects, preventing the shipment of faulty products to customers. High precision minimises false positives, reducing rework and waste and enhancing production process efficiency and profitability. The F1-Score reflects the balance between recall and precision. Furthermore, this approach that uses threshold moving allows the company to adjust its threshold according to specific needs, prioritising Precision to reduce FP or Recall to reduce FN.

These results show that incorporating an ML model into the lot-release process, as suggested by the proposed framework, can reduce customer complaints, related quality costs and enhance efficiency.

6. Conclusion

The transition to Q4.0 requires the integration of IoT, ML and Big Data to improve efficiency, productivity and innovation. Customer complaints can significantly impact automotive companies. Proactive management and anticipation of issues before shipping can boost customer satisfaction, cost savings and increase profitability.

To address this, our study focusses on improving the lot-release decision process by predicting customer complaints for an automotive company transitioning to Q4.0. We propose a framework with an ML model that gathers production line data to predict complaints. Four ML models were developed, tuned, evaluated and compared. To deal with imbalanced datasets, cost-sensitive learning and threshold-moving approaches were considered. The F1-Score and MCC were used as the main metrics to evaluate the results.

The results highlight ML's potential in refining the lot-release decision process, predicting customer complaints and reducing quality costs. XGB outperformed other algorithms with an F1-Score of 72.4% and an MCC of 75%, balancing recall and precision effectively. Compared to the heuristic rule-based system, the framework detected a significant number of tests related to customer complaints, showcasing ML's benefits. The proposed framework, using a non-sampling, threshold-moving approach, allows the prioritisation of false negatives or false positives reduction according to company-specific needs. This enables efficient model adjustment by modifying the threshold, avoiding retraining. It improves quality performance, reduces false negatives or positives and increases operational efficiency by saving time, resources and costs. The study reaffirms the importance of big data handling, improved prescriptive analytics, a change-receptive culture and top management commitment to Q4.0. We believe that our study contributes to the field of Q4.0 by underscoring the potential of ML in transforming industry practices and enhancing quality control in the era of digital transformation.

From a theoretical perspective, this study enriches the understanding of how ML can be applied to improve quality control in the context of Q4.0. It also provides insights into the importance of model evaluation metrics in the context of predicting customer complaints. Additionally, it contributes to the theoretical knowledge of handling imbalanced datasets using a non-sampling approach.

From a managerial perspective, these findings can guide strategic ML and big data investment decisions. They also stress fostering a change-receptive culture, securing top management’s Q4.0 commitment and highlighting the need for resource allocation and employee upskilling.

Leveraging a data-driven approach, a key aspect of Q4.0, the proposed framework holds several implications for the automotive industry. Contribute to reducing 0 km defects and customer complaints by preemptively identifying quality issues and enabling proactive quality control measures. This proactive approach may lead to decreased warranty and recall costs. Ultimately, it can contribute to cost savings, improved customer satisfaction and enhanced brand reputation, helping the Q4.0 transition.

Whilst the proposed framework has demonstrated promising results with real-world data, future research should replicate these findings in larger datasets for better generalisation. Exploration of alternative ML algorithms, including deep learning models, and an expanded hyperparameter search space is warranted. Additionally, resampling methods should be considered for addressing imbalanced datasets. Subsequent evaluation should comprehensively assess the framework's performance, potentially leading to the exclusive adoption of the ML approach and elimination of manual rules. Further research into the economic implications, contingent upon financial data availability, could elucidate potential cost savings and efficiency gains. The absence of financial data at this phase constrained our ability to evaluate this aspect.

Overall, this study contributes to the understanding of how advanced technologies, such as ML, can be used to improve lot-release decision processes and enhance Q4.0 adoption. The proposed framework incorporates an ML model with a non-sampling approach into the lot-release decision process and demonstrates its effectiveness in predicting customer complaints and reducing quality costs. The insights gained from this study are valuable for companies seeking to evolve towards Q4.0.

Figures

Figure 1

Evolution of quality costs distribution

Figure 2

Framework to improve the lot-release decision process

Figure 3

CRISP-DM methodology used to implement the project

Figure 4

Optimal threshold determination using Precision-Recall Curve with F1-Score

Table 1

Some features of interest in the original dataset

Field id	Description
product	Product ID number (type of product)
serial	Unique ID of a product
stationid	Unique station ID
gof status	Result of test sequence (GOF- “Good or Fail”)
cause classid	Type of defect fixed in repair
cause_flawid	Flaw identification fixed in repair

Source(s): Table by authors

Table 2

Hyperparameter searching space

Hyerparameter	RF	XGB	LGBM	CatB
n_estimators	{100,200,300}	{70,80,90,100}	{70,80,90,100}	{70,80,90,100}
max_depth	{10,20,30}	{3,4,5}	{3,4,5}	{3,4,5}
learning_rate	–	[0.1,0.3]	[0.1,0.3]	[0.1,0.3]
colsample_bytree	–	{0.5,0.6,0.7,0.8.,0.9,1}	{0.5,0.6,0.7,0.8.,0.9,1}	–
min_samples_leaf	{1,2,4}	–	–	–
in_samples_split	{2,4,6}	–	–	–

Note(s): RF: Random Forest; XGB: XGBoosst; LGBM: LightGBM; CatB: CatBoost

Source(s): Table by authors

Table 3

Best hyperparameters found using hyperopt

Hyerparameter	RF	XGB	LGBM	CatB
n_estimators	100	100	100	90
max_depth	30	5	3	5
learning_rate	–	0.2200	0.1168	0.1298
colsample_bytree	–	0.5	0.7	–
min_samples_leaf	1	–	–	–
min_samples_split	2	–	–	–

Note(s): RF Random Forest; XGB: XGBoost; LGBM: LightGBM; CatB: CatBoost

Source(s): Table by authors

Table 4

Obtained results for conceived models

Model	Precision	Recall	F1-score	MCC	FP	FN
CatB	66.7%	67.1%	66.9%	66.6%	545	536
XGB	99.5%	56.9%	72.4%	75.0%	5	703
LGBM	45.9%	46.2%	46.0%	45.5%	888	877
RF	94.1%	41.2%	57.3%	62.1%	42	958

Note(s): RF Random Forest; XGB: XGBoost; LGBM: LightGBM; CatB: CatBoost

MCC- Matthews Correlation Coefficient; FP: False Positives; FN: False Negatives

Source(s): Table by authors

Disclosure statement: The authors report that there are no competing interests to declare.