Search results

The purpose of this paper is to conduct a comprehensive review of the noteworthy contributions made in the area of the Feedforward neural network (FNN) to improve its generalization performance and convergence rate (learning speed); to identify new research directions that will help researchers to design new, simple and efficient algorithms and users to implement optimal designed FNNs for solving complex problems; and to explore the wide applications of the reviewed FNN algorithms in solving real-world management, engineering and health sciences problems and demonstrate the advantages of these algorithms in enhancing decision making for practical operations.

The FNN has gained much popularity during the last three decades. Therefore, the authors have focused on algorithms proposed during the last three decades. The selected databases were searched with popular keywords: “generalization performance,” “learning rate,” “overfitting” and “fixed and cascade architecture.” Combinations of the keywords were also used to get more relevant results. Duplicated articles in the databases, non-English language, and matched keywords but out of scope, were discarded.

The authors studied a total of 80 articles and classified them into six categories according to the nature of the algorithms proposed in these articles which aimed at improving the generalization performance and convergence rate of FNNs. To review and discuss all the six categories would result in the paper being too long. Therefore, the authors further divided the six categories into two parts (i.e. Part I and Part II). The current paper, Part I, investigates two categories that focus on learning algorithms (i.e. gradient learning algorithms for network training and gradient-free learning algorithms). Furthermore, the remaining four categories which mainly explore optimization techniques are reviewed in Part II (i.e. optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks and metaheuristic search algorithms). For the sake of simplicity, the paper entitled “Machine learning facilitated business intelligence (Part II): Neural networks optimization techniques and applications” is referred to as Part II. This results in a division of 80 articles into 38 and 42 for Part I and Part II, respectively. After discussing the FNN algorithms with their technical merits and limitations, along with real-world management, engineering and health sciences applications for each individual category, the authors suggest seven (three in Part I and other four in Part II) new future directions which can contribute to strengthening the literature.

The FNN contributions are numerous and cannot be covered in a single study. The authors remain focused on learning algorithms and optimization techniques, along with their application to real-world problems, proposing to improve the generalization performance and convergence rate of FNNs with the characteristics of computing optimal hyperparameters, connection weights, hidden units, selecting an appropriate network architecture rather than trial and error approaches and avoiding overfitting.

This study will help researchers and practitioners to deeply understand the existing algorithms merits of FNNs with limitations, research gaps, application areas and changes in research studies in the last three decades. Moreover, the user, after having in-depth knowledge by understanding the applications of algorithms in the real world, may apply appropriate FNN algorithms to get optimal results in the shortest possible time, with less effort, for their specific application area problems.

The existing literature surveys are limited in scope due to comparative study of the algorithms, studying algorithms application areas and focusing on specific techniques. This implies that the existing surveys are focused on studying some specific algorithms or their applications (e.g. pruning algorithms, constructive algorithms, etc.). In this work, the authors propose a comprehensive review of different categories, along with their real-world applications, that may affect FNN generalization performance and convergence rate. This makes the classification scheme novel and significant.

The purpose of this paper is three-fold: to review the categories explaining mainly optimization algorithms (techniques) in that needed to improve the generalization performance and learning speed of the Feedforward Neural Network (FNN); to discover the change in research trends by analyzing all six categories (i.e. gradient learning algorithms for network training, gradient free learning algorithms, optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks, metaheuristic search algorithms) collectively; and recommend new research directions for researchers and facilitate users to understand algorithms real-world applications in solving complex management, engineering and health sciences problems.

The FNN has gained much attention from researchers to make a more informed decision in the last few decades. The literature survey is focused on the learning algorithms and the optimization techniques proposed in the last three decades. This paper (Part II) is an extension of Part I. For the sake of simplicity, the paper entitled “Machine learning facilitated business intelligence (Part I): Neural networks learning algorithms and applications” is referred to as Part I. To make the study consistent with Part I, the approach and survey methodology in this paper are kept similar to those in Part I.

Combining the work performed in Part I, the authors studied a total of 80 articles through popular keywords searching. The FNN learning algorithms and optimization techniques identified in the selected literature are classified into six categories based on their problem identification, mathematical model, technical reasoning and proposed solution. Previously, in Part I, the two categories focusing on the learning algorithms (i.e. gradient learning algorithms for network training, gradient free learning algorithms) are reviewed with their real-world applications in management, engineering, and health sciences. Therefore, in the current paper, Part II, the remaining four categories, exploring optimization techniques (i.e. optimization algorithms for learning rate, bias and variance (underfitting and overfitting) minimization algorithms, constructive topology neural networks, metaheuristic search algorithms) are studied in detail. The algorithm explanation is made enriched by discussing their technical merits, limitations, and applications in their respective categories. Finally, the authors recommend future new research directions which can contribute to strengthening the literature.

The FNN contributions are rapidly increasing because of its ability to make reliably informed decisions. Like learning algorithms, reviewed in Part I, the focus is to enrich the comprehensive study by reviewing remaining categories focusing on the optimization techniques. However, future efforts may be needed to incorporate other algorithms into identified six categories or suggest new category to continuously monitor the shift in the research trends.

The authors studied the shift in research trend for three decades by collectively analyzing the learning algorithms and optimization techniques with their applications. This may help researchers to identify future research gaps to improve the generalization performance and learning speed, and user to understand the applications areas of the FNN. For instance, research contribution in FNN in the last three decades has changed from complex gradient-based algorithms to gradient free algorithms, trial and error hidden units fixed topology approach to cascade topology, hyperparameters initial guess to analytically calculation and converging algorithms at a global minimum rather than the local minimum.

The existing literature surveys include comparative study of the algorithms, identifying algorithms application areas and focusing on specific techniques in that it may not be able to identify algorithms categories, a shift in research trends over time, application area frequently analyzed, common research gaps and collective future directions. Part I and II attempts to overcome the existing literature surveys limitations by classifying articles into six categories covering a wide range of algorithm proposed to improve the FNN generalization performance and convergence rate. The classification of algorithms into six categories helps to analyze the shift in research trend which makes the classification scheme significant and innovative.

The purpose of this study is to propose a framework for expedited antenna optimization with numerical derivatives involving gradient variation monitoring throughout the optimization run and demonstrate it using a benchmark set of real-world wideband antennas. A comprehensive analysis of the algorithm performance involving multiple starting points is provided. The optimization results are compared with a conventional trust-region (TR) procedure, as well as the state-of-the-art accelerated TR algorithms.

The proposed algorithm is a modification of the TR gradient-based algorithm with numerical derivatives in which a monitoring of changes of the system response gradients is performed throughout the algorithm run. The gradient variations between consecutive iterations are quantified by an appropriately developed metric. Upon detecting stable patterns for particular parameter sensitivities, the costly finite differentiation (FD)-based gradient updates are suppressed; hence, the overall number of full-wave electromagnetic (EM) simulations is significantly reduced. This leads to considerable computational savings without compromising the design quality.

Monitoring of the antenna response sensitivity variations during the optimization process enables to detect the parameters for which updating the gradient information is not necessary at every iteration. When incorporated into the TR gradient-search procedures, the approach permits reduction of the computational cost of the optimization process. The proposed technique is dedicated to expedite direct optimization of antenna structures, but it can also be applied to speed up surrogate-assisted tasks, especially solving sub-problems that involve performing numerous evaluations of coarse-discretization models.

The introduced methodology opens up new possibilities for future developments of accelerated antenna optimization procedures. In particular, the presented routine can be combined with the previously reported techniques that involve replacing FD with the Broyden formula for directions that are satisfactorily well aligned with the most recent design relocation and/or performing FD in a sparse manner based on relative design relocation (with respect to the current search region) in consecutive algorithm iterations.

Benchmarking against a conventional TR procedure, as well as previously reported methods, confirms improved efficiency and reliability of the proposed approach. The applications of the framework include direct EM-driven design closure, along with surrogate-based optimization within variable-fidelity surrogate-assisted procedures. To the best of the authors’ knowledge, no comparable approach to antenna optimization has been reported elsewhere. Particularly, it surmounts established methodology by carrying out constant supervision of the antenna response gradient throughout successive algorithm iterations and using gathered observations to properly guide the optimization routine.

Inspired by the basic idea of gradient boosting, this study aims to design a novel multivariate regression ensemble algorithm RegBoost by using multivariate linear regression as a weak predictor.

To achieve nonlinearity after combining all linear regression predictors, the training data is divided into two branches according to the prediction results using the current weak predictor. The linear regression modeling is recursively executed in two branches. In the test phase, test data is distributed to a specific branch to continue with the next weak predictor. The final result is the sum of all weak predictors across the entire path.

Through comparison experiments, it is found that the algorithm RegBoost can achieve similar performance to the gradient boosted decision tree (GBDT). The algorithm is very effective compared to linear regression.

This paper attempts to design a novel regression algorithm RegBoost with reference to GBDT. To the best of the knowledge, for the first time, RegBoost uses linear regression as a weak predictor, and combine with gradient boosting to build an ensemble algorithm.

This article proposes a relaxed gradient iterative (RGI) algorithm to solve coupled Sylvester-conjugate transpose matrix equations (CSCTME) with two unknowns.

This article proposes a RGI algorithm to solve CSCTME with two unknowns.

The introduced (RGI) algorithm is more efficient than the gradient iterative (GI) algorithm presented in Bayoumi (2014), where the author's method exhibits quick convergence behavior.

The introduced (RGI) algorithm is more efficient than the GI algorithm presented in Bayoumi (2014), where the author's method exhibits quick convergence behavior.

In systems and control, Lyapunov matrix equations, Sylvester matrix equations and other matrix equations are commonly encountered.

In systems and control, Lyapunov matrix equations, Sylvester matrix equations and other matrix equations are commonly encountered.

This article proposes a relaxed gradient iterative (RGI) algorithm to solve coupled Sylvester conjugate transpose matrix equations (CSCTME) with two unknowns. For any initial matrices, a sufficient condition is derived to determine whether the proposed algorithm converges to the exact solution. To demonstrate the effectiveness of the suggested method and to compare it with the gradient-based iterative algorithm proposed in [6] numerical examples are provided.

Introduces papers from this area of expertise from the ISEF 1999 Proceedings. States the goal herein is one of identifying devices or systems able to provide prescribed performance. Notes that 18 papers from the Symposium are grouped in the area of automated optimal design. Describes the main challenges that condition computational electromagnetism’s future development. Concludes by itemizing the range of applications from small activators to optimization of induction heating systems in this third chapter.

This paper aims to effectively explore the application effect of big data techniques based on an α-support vector machine-stochastic gradient descent (SVMSGD) algorithm in third-party logistics, obtain the valuable information hidden in the logistics big data and promote the logistics enterprises to make more reasonable planning schemes.

In this paper, the forgetting factor is introduced without changing the algorithm's complexity and proposed an algorithm based on the forgetting factor called the α-SVMSGD algorithm. The algorithm selectively deletes or retains the historical data, which improves the adaptability of the classifier to the real-time new logistics data. The simulation results verify the application effect of the algorithm.

With the increase of training times, the test error percentages of gradient descent (GD) algorithm, gradient descent support (SGD) algorithm and the α-SVMSGD algorithm decrease gradually; in the process of logistics big data processing, the α-SVMSGD algorithm has the efficiency of SGD algorithm while ensuring that the GD direction approaches the optimal solution direction and can use a small amount of data to obtain more accurate results and enhance the convergence accuracy.

The threshold setting of the forgetting factor still needs to be improved. Setting thresholds for different data types in self-learning has become a research direction. The number of forgotten data can be effectively controlled through big data processing technology to improve data support for the normal operation of third-party logistics.

It can effectively reduce the time-consuming of data mining, realize the rapid and accurate convergence of sample data without increasing the complexity of samples, improve the efficiency of logistics big data mining, reduce the redundancy of historical data, and has a certain reference value in promoting the development of logistics industry.

The classification algorithm proposed in this paper has feasibility and high convergence in third-party logistics big data mining. The α-SVMSGD algorithm proposed in this paper has a certain application value in real-time logistics data mining, but the design of the forgetting factor threshold needs to be improved. In the future, the authors will continue to study how to set different data type thresholds in self-learning.

In light of the current international concerns with security and terrorism, interest is increasing on the topic of using robot swarms to locate the source of chemical hazards. The purpose of this paper is to place this task, called chemical plume tracing (CPT), in the context of fluid dynamics.

This paper provides a foundation for CPT based on the physics of fluid dynamics. The theoretical approach is founded upon source localization using the divergence theorem of vector calculus, and the fundamental underlying notion of the divergence of the chemical mass flux. A CPT algorithm called fluxotaxis is presented that follows the gradient of this mass flux to locate a chemical source emitter.

Theoretical results are presented confirming that fluxotaxis will guide a robot swarm toward chemical sources, and away from misleading chemical sinks. Complementary empirical results demonstrate that in simulation, a swarm of fluxotaxis‐guided mobile robots rapidly converges on a source emitter despite obstacles, realistic vehicle constraints, and flow regimes ranging from laminar to turbulent. Fluxotaxis outperforms the two leading competitors, and the theoretical results are confirmed experimentally. Furthermore, initial experiments on real robots show promise for CPT in relatively uncontrolled indoor environments.

A physics‐based approach is shown to be a viable alternative to existing mainly biomimetic approaches to CPT. It has the advantage of being analyzable using standard physics analysis methods.

The fluxotaxis algorithm for CPT is shown to be “correct” in the sense that it is guaranteed to point toward a true source emitter and not be fooled by fluid sinks. It is experimentally (in simulation), and in one case also theoretically, shown to be superior to its leading competitors at finding a source emitter in a wide variety of challenging realistic environments.

Discrete control of robot manipulators with uncertain model is the purpose of this paper.

The proposed control design is model-free by employing an adaptive fuzzy estimator in the controller for the estimation of uncertainty as unknown function. An adaptive mechanism is proposed in order to overcome uncertainties. Parameters of the fuzzy estimator are adapted to minimize the estimation error using a gradient descent algorithm.

The proposed model-free discrete control is robust against all uncertainties associated with the model of robotic system including the robot manipulator and actuators, and external disturbances. Stability analysis verifies the proposed control approach. Simulation results show its efficiency in the tracking control.

A novel model-free discrete control approach for electrically driven robot manipulators is proposed. An adaptive fuzzy estimator is used in the controller to overcome uncertainties. The parameters of the estimator are regulated by a gradient descent algorithm. The most gradient descent algorithms have used a known cost function based on the tracking error for adaptation whereas the proposed gradient descent algorithm uses a cost function based on the uncertainty estimation error. Then, the uncertainty estimation error is calculated from the joint position error and its derivative using the closed-loop system.

The purpose of this paper is to study the optimization problem of low‐frequency magnetic shielding using the adjoint variable method (AVM). This method is compared with conventional methods to calculate the gradient.

The equation for the vector potential (eddy currents model) in appropriate Sobolev spaces is studied to obtain well‐posedness. The optimization problem is formulated in terms of a cost functional which depends on the vector potential and its rotation. Convergence of a steepest descent algorithm to a stationary point of this functional is proved. Finally, some numerical results for an axisymmetric induction heater are presented.

Using Friedrichs' inequality, the existence and uniqueness of the vector potential, its gradient and the corresponding adjoint variable can be proved. From the numerical results, it is concluded that the AVM is advantageous if the number of parameters to optimize is larger than two.

The AVM is only faster than conventional methods if the gradients can be calculated with sufficient accuracy.

Theoretical results for eddy currents model are often based on a non‐vanishing conductivity. The theoretical value of this paper is the presence of non‐conducting materials in the domain. From a practical viewpoint, it has been demonstrated that the AVM can yield a significant reduction of computational time for advanced optimization problems.