Detection of solder paste defects with an optimization‐based deep learning model using image processing techniques

Abstract

Purpose

In the production processes of electronic devices, production activities are interrupted due to the problems caused by soldering defects during the assembly of surface-mounted elements on printed circuit boards (PCBs), and this leads to an increase in production costs. In solder paste applications, defects that may occur in electronic cards are usually noticed at the last stage of the production process. This situation reduces the efficiency of production and causes delays in the delivery schedule of critical systems. This study aims to overcome these problems, optimization based deep learning model has been proposed by using 2D signal processing methods.

Design/methodology/approach

An optimization-based deep learning model is proposed by using image-processing techniques to detect solder paste defects on PCBs with high performance at an early stage. Convolutional neural network, one of the deep learning methods, is trained using the data set obtained for this study, and pad regions on PCB are classified.

Findings

A total of six types of classes used in the study consist of uncorrectable soldering, missing soldering, excess soldering, short circuit, undefined object and correct soldering, which are frequently used in the literature. The validity of the model has been tested on the data set consisting of 648 test data.

Originality/value

The effect of image processing and optimization methods on model performance is examined. With the help of the proposed model, defective solder paste areas on PCBs are detected, and these regions are visualized by taking them into a frame.

Citation

Sezer, A. and Altan, A. (2021), "Detection of solder paste defects with an optimization‐based deep learning model using image processing techniques", Soldering & Surface Mount Technology, Vol. 33 No. 5, pp. 291-298. https://doi.org/10.1108/SSMT-04-2021-0013

1. Introduction

Printed circuit boards (PCBs) used in electronic devices are produced as surface mounted depending on the elements used. To attach electronic components to the copper pad surfaces, solder paste application is carried out in accordance with mass production by means of devices designed for this purpose. In this process, production may be interrupted because of defects caused by the solder paste and pad surfaces, or the tests of the manufactured board may fail because of the failure to successfully mount the circuit elements on the PCB (Hao et al., 2013; Mohanty et al., 2008; Hassanin et al., 2019). To overcome these problems, an optimization-based convolutional neural network (CNN) model has been developed using 2 D signal processing methods.

The soldering defects frequently encountered on PCBs during soldering can be listed as solder bridging, lifted components (tombstone), excessive solder, solder balling, solder wetting/non-wetting, solder skips, solder flag and solder discoloration (Mohanty et al., 2008; Hassanin et al., 2019; Hua et al., 2018). The defects encountered because of incorrect application of solder paste can be listed as common and rare defects. Solder bridging, lifted component, excessive solder and solder balling are common soldering faults. The rarer soldering defects, compared to common soldering defects, are solder skips, solder flag, solder discoloration and solder wetting/non-wetting.

The mentioned soldering defects are the faults encountered at the end of the production process and often observed because of not noticing faulty solder paste application, which takes a long time to overcome. Even if these defects are corrected after the production process, these PCBs are considered defective in terms of production quality (Ozturk and Akdemir, 2017). PCBs with soldering defects are called faulty production and are not used because of the possibility of problems later on in the manufactured devices. This situation causes uneconomical use of time and resources and delays production. Even if precautions are taken by subjecting the determined number of PCBs to this process to prevent the losses before each production, solder paste defects in a single PCB, which is produced for a special purpose, may cause serious time and resource losses in some special cases (Chauhan and Bhardwaj, 2011).

There are some studies in the literature to detect defects on PCBs. In the study conducted by Hao et al. (2013), they aimed to observe solder connection defects seen in surface mount components on PCBs and increase the performance of automatic optical inspection by combining a genetic algorithm and a neural network. In their study, six types of defects were used, and a 98.46% success was achieved by combining back propagation neural network and genetic algorithm. In the study by Mohanty et al. (2008), 2 D and 3 D area-volume connections of small-sized components have been studied and it was aimed at providing an investment return on PCB solder paste examination. In the study, which started with the widespread use of small-sized components on PCBs and the use of lead-free solder to solder them more efficiently, the connection between 2 D area measurements of components of different sizes and the height map derived from the area created because of solder paste inspection was examined. It was seen that the 2 D method provided higher field data than the 3 D method, and linear regression was used to understand the limits of estimation and link estimation between both. Hassanin et al. (2019) examined all defects in PCBs, and these defects were classified using an image processing technique. In the study created by collecting a total of 14 classes in five categories, the defects and differences in the visual data were first processed by dividing the PCB images into four regions and new images were obtained. Each new image acquired was classified by image processing. The regions of the defects found on the PCB were specified. Huang and Wei (2018) tried to detect defects on a PCB by using the CNN fed with 1386 PCBs. They drew attention to the data set problem experienced in studies conducted with neural networks and published their own data. There were six basic soldering defects in the data sets. With the deep learning method, they tried to eliminate the efficiency problems encountered in previous studies and achieved an average success of 99%. In the study, the defects were detected and classified using the connection boxes for each defect. Chauhan and Bhardwaj (2011) examined the PCB production, component control and soldering stages in their study, and the defects of these stages were obtained by taking visual difference from the original image with the help of logic operations. The control of the boards for all three stages enabled the defects to be determined separately for each PCB production, component and soldering stages, and the defects found were displayed in colour and the user was informed.

In this study, different from studies in the literature, an optimization-based CNN model is proposed by using 2 D signal processing methods to detect solder defects on PCBs. The performance of the proposed model is quite durable and higher than the models in the literature. In the developed model, the data set prepared for the classifier was divided into six classes as determined soldering defects and correct application and presented to the user by visualization. The rest of the study is organized as follows. In Section 2, data collection and data pre-processing are presented. In Section 3, the application of CNN to the study with 2 D signal processing algorithms and optimization techniques is described. In Section 4, the results obtained with the proposed method are presented and these results are compared with other methods in the literature. Finally in Section 5, the results are highlighted.

2. Data collection and pre-processing

2.1 Collection of data

To detect and avoid solder paste defects on PCBs, frequently occurring defects on the boards were determined. The data set including the determined defect classes was created using the experimental setup in Figure 1(a). The defects encountered in solder paste applications applied on PCBs are separated into six basic groups as; uncorrectable soldering, missing soldering, excess soldering, short circuit, undefined object and correct soldering. Trial PCBs containing samples belonging to each group were prepared. Images of the trial PCBs were taken with a 3840 (H) × 2160 (V) resolution camera. The data set was created using a total of four different PCBs. Because of the high resolution of each image in the prepared data set, the processing time took considerably longer. To overcome this problem, a new data set was created by obtaining the templates of the defective regions in the images (Yoo et al., 2020; Hossin and Sulaiman, 2015). Also, as solder defects have been focused on in the study, thanks to the templates obtained, the waterways that can be seen clearly under the coating on the PCB surface have been separated. Thus, a data set that could be used for training the neural network was obtained.

2.2 Preparation of data for neural network model

Large data sets are crucial for system performance and stability in deep learning models. The performance of the model increases significantly with the size and variety of the data set (Yoo et al., 2020). Therefore, the data set prepared for this study was expanded. The images were made up of pixels. Each image had height and width pixels, and each pixel was a cell that contained the numerical value of the relevant area. From this perspective, each image was actually a matrix of pixels. In light of this information, the expansion process was carried out using signal processing techniques (Park et al., 2020). The techniques of flipping, rotating, shifting, adding noise and blurring the image were used (Mikołajczyk and Grochowski, 2018).

2.2.1 Flipping

It was mentioned above that the images, i.e. visual signals, were made up of matrices. In this direction, matrix operations were used to process the images horizontally and vertically. Turning an image on a horizontal or vertical axis is one of the methods frequently used in data derivation techniques. The ability to flip a horizontal or vertical axis is performed by reversing the values of each column or row in the matrix of the image (Gómez-Ríos et al., 2019). In this way, a copy of the original image rotated around the x or y axes was obtained.

2.2.2 Rotation

The rotation process was performed by the command “rotate()” in the skimage library in Python language. In order for the command to work, an image to be rotated and angle value must be entered. If the angle was selected positive, the image was rotated counterclockwise; if the angle was negative, the image was rotated clockwise. The data set was expanded by rotating the data used at −45 and +45 degrees.

2.2.3 Shifting

Images can be shifted in the desired destination in the direction of the specified value. It is necessary to determine the displacement values on the horizontal and vertical axis to perform the shifting process. With these values, the starting position was shifted to a new point. If x and y were assigned to the initial position values and dx and dy to the displacements, the new position obtained because of the shifting process was the values of X and Y. Mathematically expressed as X = x + dx and Y = y + dy.

2.2.4 Adding noise

This technique adds random noise to the image and is often used to edit an image. It prevents the model from falling into overfitting in deep learning applications. It was used with the help of the “random_noise ()” command from the skimage library in Python language.

2.2.5 Blurring image

Gaussian blurring in image processing is the result of blurring an image with the Gaussian function. It is mostly used in graphic software to reduce the details and remove unwanted noise on the image. Gaussian correction is also used as a preprocessing step in computer vision algorithms to develop image structures at different scales. The visual effect of this blur technique is a smooth blur, similar to viewing the image on a translucent screen, distinctly different from the bokeh effect produced by an out-of-focus lens or the shadow of an object under normal lighting (Jiang et al., 2012). The obtained data set was arranged in a proportional way to avoid problems such as overfeeding during the training process by deriving new images from the training data set.

3. Method

In this section, the CNN model used for the classifier is explained. It is widely used in areas such as segmentation and classification, due to its ability to decompose features of images (Jiang et al., 2012). Using less preprocessing makes this algorithm more useful than other image classification algorithms. The CNN algorithms have many different architectures within themselves (Yoo et al., 2020). In this study, CNN was used together with the rectified linear unit (ReLu) activation function, and its structure is presented in Figure 2. The training data obtained through the image data processed for the operation of CNN was sent to the entrance of the neural network. The input class was obtained by processing the input data through convolution, activation, pooling, flattening, full link and classification layers. The framework of the proposed model for detecting solder paste defects is presented in Figure 3.

3.1 Convolution layer

Convolutional is the basic layer of the artificial neural network. The input image presented to this layer is treated as a matrix. In the layer, the properties of the image are obtained using a filter. Generally, filters can be selected in 3 × 3 or 5 × 5 sizes. As a result, it reveals the feature map of the image (Yoo et al., 2020).

3.2 Activation layer

The activation layer is the layer that subjects the convolution layer outputs to nonlinear processing. For this study, the ReLu activation function was chosen from the layer that had various activation functions. The ReLu activation function assigns negative values to zero instead of processing negative values. That is, although the result is zero for values less than zero, it returns its own value as a result for values greater than zero.

3.3 Pooling layer

It is the layer that performs by reducing the slip size, network parameters and the number of calculations that can be encountered on the processed data to determine the incompatibilities that may occur on the neural network. In this study, maxpooling algorithm (Gómez-Ríos et al., 2019), which is the most popular pooling process, was used. Maxpooling uses a filter applied on the image to select the largest value as output and uses smaller outputs containing meaningful information in order for the neural network to reach the correct decision.

3.4 Flattening layer

It is the layer that creates and prepares the input data for the full link layer. The data in this layer are the forms of the data taken from the convolution layer and pooling layer transformed into a one-dimensional array (Hossin and Sulaiman, 2015).

3.5 Full link layer

In this layer, data from the flattening layer in the form of a one-dimensional matrix is processed. The connection between neurons is sequentially linked, taking into account the data presented as input, and the final properties are selected for classification (Yoo et al., 2020).

3.6 Classification layer

In the classification layer, the class is determined using various classifiers. Although softmax is mostly preferred as the classifier, binary cross entropy was preferred as the classifier as two classes were used in this study. Binary cross entropy is a logarithmic loss function that responds to the question posed with only two options as true or false, x or y. It is classified as true and false as the output of this logarithmic function (Yoo et al., 2020).

3.7 Optimization method

Optimization is the method used to make learning more efficient by minimizing the difference between the output value produced by the network and the actual value. In this study, the efficiency of the system was compared using Adam, RMSprop and AdaMax algorithms as the optimization method (Kingma and Ba, 2017).

Adam optimization is an optimization method based on gradient descent. It is used to improve efficiency in the model and its mathematical expression is as follows:

(1) mt=β1mt−1+(1−β1)gt

(2) vt=β2vt−1+(1−β2)gt2

(3) m′t=mt1−β1t , v′t=vt1−β2t

(4) θt+1=θt−ηv′t+ϵm′t

where η is step size. β₁ and β₂ indicate exponential decay rates for the moment estimates, and these should be selected between 0 and 1. v_t is exponentially weighted averages of squares of historical slopes, as well as m_t caching momentum changes. θ_t is the initial parameter vector. g_t is stochastic objective at time step t, ϵ is a small constant for numerical stability (10⁻⁸), and u_t is the exponentially weighted infinity norm.

This optimization converges to the target faster by eliminating the oscillations encountered during the search for the optimum point to increase efficiency. By taking the squares of the momentum gradients of the learning coefficient value, it can overcome the constant coefficient problem.

RMSprop optimization is also based on the gradient descent optimization method. The RMSprop optimization method has been introduced with the aim of providing a solution to the problem of excessive reduction of learning coefficient. To solve the problem, instead of using the values obtained from all the slope squares obtained in previous trainings, its amount is limited to a certain frame size. This method operates with a different learning coefficient at each step to overcome the excessive drop in learning coefficient. The learning coefficient was updated by dividing by the square root of the cumulative sums of squares of past momentum gradients. The mathematical expression of RMSprop optimization is as follows:

(5) E[g2]t=0.9E[g2]t−1+0.1gt2

(6) θt+1=θt−ηE[g2]t+ϵgt

(7) gt=∇θtJ(θt)

where η is step size; θ_t is the initial parameter vector; g_t is stochastic objective at time step t; ϵ is a small constant for numerical stability (10⁻⁸); E is expected value; and J refers to the objective function that should be minimized.

AdaMax optimization is also based on the gradient descent optimization method. AdaMax is a variant of the Adam optimization method based on infinity norm. AdaMax optimization can be superior to Adam optimization, especially in comparison models. Unlike Adam, the sparse implementation of this algorithm only updates the variable slices and the corresponding m_t, v_t terms when that part of the variable is used in the forward transition. This means that sparse behavior is the opposite of intense behavior. The mathematical expressin of AdaMax optimization is modeled as:

(9) vt=β2vt−1+(1−β2)|gt|2

(10) vt=β2pvt−1+(1−β2p)|gt|p

(11) ut=β2∞vt−1+(1−β2∞)|gt|∞     =max(β2⋅vt−1, |gt|)

(12) θt+1=θt−ηutm′t

where v_t is exponentially weighted averages of squares of historical slopes, as well as m_t caching momentum changes. η is step size. β₁ and β₂ indicate the exponential decay rates for the moment estimates, and these should be selected between 0 and 1. θ_t is the initial parameter vector; g_t is stochastic objective at time step t; and u_t is the exponentially weighted infinity norm (Kingma and Ba, 2017).

3.8 Image processing

It is a signal processing method whose input is an image or video data and aims to obtain the desired information with the help of signal processing techniques on the presented data. The data set, which is divided into true and false classes as a result of the classifier, is determined on the original image with the help of image processing techniques, and the positions of the neural network model outputs can be determined. To determine the position of the template on the original image, the template width and height dimensions are determined, and a matching process is made with the help of these values, as seen Figure 4.

4. Experimental results and discussion

In this section, the processing of the data set prepared with the methods mentioned earlier is discussed. First, the data processed by image processing techniques were prepared to feed the CNN, and the data obtained by using the neural network were processed with image processing techniques. The data set was separated into two classes with CNN and separated into six subcategories by image processing methods. In addition, the efficiency of the result was compared with Adam, RMSprop and AdaMax optimization methods, and the most efficient system was tried to be obtained.

4.1 Solder paste data set

In the study, the data set was prepared by collecting images of the cards with solder paste in 3840 × 2160 resolution. A total of 274 raw card images were transformed into smaller sized raw data to overcome long processing times and insufficient system requirements that can be experienced in image processing and neural network training because of its high resolution. A data set was obtained for correct and incorrect applications from the pad regions to be classified. In order for the system to work more efficiently, each image in the data set was resized, and the pixel value of each image was adjusted to 50 × 50. Data set examples are shown in Figure 5.

The training data set was prepared to feed the neural network by reproducing the data set with the techniques given under the heading of data collection and pre-processing. 291 correct and 291 incorrect images data were used to feed the neural network. The test data allocated to test the training process consisted of 33 correct and 33 incorrect images, as can be seen in Table 1. The system was trained and tested with a total of 648 data.

Identifying the incorrectly classified samples on the card was achieved by matching the data set prepared for training with image processing. For this purpose, the prepared solder paste data set was rearranged as correct application and incorrect application types, and pad classes were created. The classes are shown in Table 2.

4.2 Performance metrics

In this study, to keep the classifier efficiency high, accuracy metric was used as the classifier metric. The confusion matrix is given Table 3, and the metric types and mathematical expressions are given in Table 4.

To briefly define the metrics given above; the accuracy measure was used in this study because it determines the proportion of correct predictions out of the total number of samples used in training. The sensitivity was used to measure positive predictions that were correct because of the classification. Specificity was used to measure negative predictions that were correct because of classification. The precision was used to measure the number positively predicted correctly among the results obtained because of the classification. The recall measured how accurately positive predictions were classified. F-measure is the harmonic mean between the recall and precision metrics.

4.3 Experimental results

In this section, the performance results obtained by using three different optimization methods of the code working with 100-epoch training are compared. First of all, the performance rates obtained for all three methods evaluated by taking the average of the accuracy results for each epoch are given in Table 5. According to the results obtained from the tables given above, it is seen that the success rate of AdaMax optimization was more efficient compared to RMSprop and Adam optimization methods.

The data obtained from each epoch process were recorded and compared on the graph in Figure 6, to see the effects of optimization methods on performance more clearly.

As can be seen from the graphics and performance values, more efficient results were obtained by using the AdaMax optimization method together with the ReLu algorithm compared to the RMSprop and Adam optimization methods. The classification results obtained by ReLu Algorithm and AdaMax optimization are matched with the original image and marked with unique colors for each class. Image matching started with the recording of the pixel positions where continuous similarity was achieved by determining the common pixels with the template on the original image. Then the classified images were indicated by square frames on the original image. Square frames were drawn using the length and height of the defect over the recorded pixel locations. An example of the application is shown in Figure 7.

5. Conclusions

Realizing the soldering defects that are faced during typesetting operations of PCBs and preventing those defects are quite significant for the efficiency of the process. In this study, to prevent incorrect production, caused by defects in the solder paste applied to the PCBs, a method based on deep learning that classifies the solder paste defects by using image processing techniques and that aims to indicate the locations of the mis-implementations on the card image has been used. The aim was to increase efficiency by noticing the mis-implementations at an early stage. Additionally, on the neural network model, the effects of optimization methods on efficiency were studied and compared. The neural network model that was suggested here was composed by using the ReLu activation function, and as a result of comparisons, this model was supported by the AdaMax optimization method that was highly efficient. By using the optimization methods to minimize the margin of defect, an increase in the resulting performance is experienced. With the data obtained, it was observed that, as compared to the RMSprop and Adam methods, the AdaMax optimization method was more efficient in achieving results with 96.40% success rate and in 100 epoch loops. As efficiency in classifiers is calculated with the ratio of correct estimates, the accuracy measurement method was used as criterion. The data set that contained 648 visual data and that was specially prepared for this research and solder paste defects were classified under two headings as true and false by feeding the CNN. Five class examples that can be enumerated as uncorrectable solder, missing solder, excess solder, short circuit, undefined object, and including correct application, six class examples were paired with the original image of the PCB by applying image processing techniques. As a result of pairings, class locations were framed. It was observed that the usage of the proposed model and the image processing methods together was quite efficient for identifying faulty soldering applications.

Figures

Figure 1

Camera used for data collection and data preparation (a) camera setup; (b) camera; (c) PCB image; and (d) extracted pad region

Figure 3

Framework of the proposed model for detecting solder defects

Figure 2

CNN structure

Figure 4

Image matching samples for detecting solder defects (a) original; (b) matched

Figure 5

Samples from collected data set (a) excess solder; (b) missing solder; (c) short circuit; and (d) undefined object

Figure 6

Accuracy comparison for each epoch of AdaMax, RMSprop and Adam optimizations

Figure 7

Solder pad locations classified by defect categories color representation, respectively blue for correct, red for uncorrectable, cyan for missing solder, green for excess solder, yellow for short circuit and purple for undefined object

Table 1

Solder paste data set

Classes	Raw data	Data	Training data	Test data	Total
Correct	4	324	291	33	324
Incorrect	270	324	291	33	324
Total	274	648	582	66	648

Table 2

Solder defects classes and samples

Types of defects	Samples
Correct	324
Uncorrectable	51
Missing solder	60
Excess solder	86
Short circuit	60
Undefined object	67

Table 3

Confusion matrix for the proposed method

Confusion matrix	Prediction		Total
	Correct	Incorrect
Test data
Correct	TP = 22	FN = 11	33
Incorrect	FP = 0	TN = 33	33
Total	22	44	66

Table 4

Performance metrics

Metrics	Formulation	Results
Accuracy	(TP + TN)/(TP + FP + TN + FN)	0.83
Sensitivity	TP/(TP + FN)	0.67
Specificity	TN/(TN + FP)	1.00
Precision	TP/(TP + FP)	1.00
Recall	TP/(TP + TN)	0.40
F-measure	2*TP/(2*TP + FP + FN)	0.80

Table 5

Experimental performance results obtained with accuracy metric

Optimization	Performance (%)	Losses (%)
AdaMax	96.40	13.56
RMSprop	94.67	32.46
Adam	95.53	30.91