Published January 28, 2007 | Version 10808

Improving the Convergence of the Backpropagation Algorithm Using Local Adaptive Techniques

Description

Since the presentation of the backpropagation algorithm, a vast variety of improvements of the technique for training a feed forward neural networks have been proposed. This article focuses on two classes of acceleration techniques, one is known as Local Adaptive Techniques that are based on weightspecific only, such as the temporal behavior of the partial derivative of the current weight. The other, known as Dynamic Adaptation Methods, which dynamically adapts the momentum factors, α, and learning rate, η, with respect to the iteration number or gradient. Some of most popular learning algorithms are described. These techniques have been implemented and tested on several problems and measured in terms of gradient and error function evaluation, and percentage of success. Numerical evidence shows that these techniques improve the convergence of the Backpropagation algorithm.

Files

10808.pdf

Files (214.2 kB)

Name Size
md5:6c7af5573f893905bdf353b3d2b9d1f6
214.2 kB Preview Download

Additional details

References

  • Magoulas, G.D., Plagianakos, V.P., & Vrahatis, M.N. Globally Convergent Algorithms with Local Learning Rates. In IEEE Transactions on Neural Networks, Vol 13, No 3, 774 - 779, 2002.
  • Silva F.M. & Almeida L.B., Acceleration techniques for the backpropagation algorithm. Neural Networks EURASIP Workshop, Sesim, 1990.
  • Jacobs R. A., Increased rates of convergence through learning rate adaptation, Neural Networks, 1(4), 295-308, 1988.
  • Tollenaere, T., SuperSab:fast adaptive backpropagation with good scaling properties. Neural Networks, 3(5), 1990.
  • Fahlman, S.E., An empirical study of learning speed in backpropagation networks, Technical Report, CMU-CS-88-162, 1988
  • Vrahatis, M.N., Magoulas, G.D. & Plagaianakos, V.P. Convergence Analysis of the Quickprop Method. University of Patras and Athens, 1999.
  • Riedmiller, M. & Braun, H., A direct adaptive method for faster backpropagation learning. The RPROP algorithm. In Proceedings of the IEEE International Conference on Neural Networks (ICNN) (Ruspini, H. ed), p. 586-591, 1993, San Francisco.
  • Evans, D.J. and Zainuddin, Z., Acceleration of the backpropagation through dynamic adaptation of the momentum. Neural, Parallel & Scientific Computations, 5(3), 1997, 297-308. (see also Internal Report No.1028, PARC, Loughborough University of Tech., U.K., 1996).
  • Zainuddin Z. & Evans D.J., Acceleration of the backpropagation through dynamic adaptation of the learning rate, Int. Journal of Computer Mathematics, 334, 1-17, 1997 (see also Internal report No. 1029, PARC, Loughborough University of Tech., U.K. 1996). [10] Zainuddin Z. & Sathasivam S. , Modeling nonlinear relationships in ecology and biology using neural networks, Proceedings of the National Workshop in Ecological and environmental modeling (ECOMOD), Sept. 3-4, 2001. [11] Zainuddin Z. & Ahmad Fadzil, M.H., Training Feedforward Neural Networks: An Algorithm Giving Improved Convergence. Proceedings of research & Development in Computer Science & its Application,p.98- 102, 1998, Penang, Malaysia.