To read this content please select one of the options below:

Multi‐directional search to optimize neural network error function

Shamsuddin Ahmed (American University of the Middle East in affiliation with Purdue University, Dasman, Kuwait Edith Cowan University, Perth, Australia)

Kybernetes

ISSN: 0368-492X

Article publication date: 10 August 2010

252

Abstract

Purpose

The proposed algorithm successfully optimizes complex error functions, which are difficult to differentiate, ill conditioned or discontinuous. It is a benchmark to identify initial solutions in artificial neural network (ANN) training.

Design/methodology/approach

A multi‐directional ANN training algorithm that needs no derivative information is introduced as constrained one‐dimensional problem. A directional search vector examines the ANN error function in weight parameter space. The search vector moves in all possible directions to find minimum function value. The network weights are increased or decreased depending on the shape of the error function hyper surface such that the search vector finds descent directions. The minimum function value is thus determined. To accelerate the convergence of the algorithm a momentum search is designed. It avoids overshooting the local minimum.

Findings

The training algorithm is insensitive to the initial starting weights in comparison with the gradient‐based methods. Therefore, it can locate a relative local minimum from anywhere of the error surface. It is an important property of this training method. The algorithm is suitable for error functions that are discontinuous, ill conditioned or the derivative of the error function is not readily available. It improves over the standard back propagation method in convergence and avoids premature termination near pseudo local minimum.

Research limitations/implications

Classifications problems are efficiently classified when using this method but the complex time series in some instances slows convergence due to complexity of the error surface. Different ANN network structure can further be investigated to find the performance of the algorithm.

Practical implications

The search scheme moves along the valleys and ridges of the error function to trace minimum neighborhood. The algorithm only evaluates the error function. As soon as the algorithm detects flat surface of the error function, care is taken to avoid slow convergence.

Originality/value

The algorithm is efficient due to incorporation of three important methodologies. The first mechanism is the momentum search. The second methodology is the implementation of directional search vector in coordinate directions. The third procedure is the one‐dimensional search in constrained region to identify the self‐adaptive learning rates, to improve convergence.

Keywords

Citation

Ahmed, S. (2010), "Multi‐directional search to optimize neural network error function", Kybernetes, Vol. 39 No. 7, pp. 1145-1166. https://doi.org/10.1108/03684921011062755

Publisher

:

Emerald Group Publishing Limited

Copyright © 2010, Emerald Group Publishing Limited

Related articles