Weight Initialization for High Order and Multilayer Perceptrons

Proper weight initialization is one of the most important prerequisites for fast convergence of feed-forward neural networks like high order and multilayer perceptrons. In order to determine the optimal value of the initial weight variance (or range), which is a important parameter of random weight initialization methods for high order perceptrons, a wide range of experiments (more than $200,000$ simulations) was performed, using seven different data sets, three weight distributions, three activation functions, and several network orders. The results of these experiments are compared to weight initialization techniques for multilayer perceptrons, which leads to the proposal of a suitable weight initialization method for high order perceptrons. Experiments over a large range of initial weight variances are performed (more than $20,000$ simulations) for multilayer perceptrons and compared to weight initialization methods proposed by other authors. The results of this comparison are justified by sufficiently small confidence intervals.

Aguilar, Marc
Published in:
Proceedings of the '94 SIPAR--Workshop on Parallel and Distributed Computing, 87-90
Presented at:
SI Group for Parallel Systems - Proceedings of the '94 SIPAR--Workshop on Parallel and Distributed Computing
Institute of Informatics University, P\'erolles, Chemin du Mus\'ee 3, CH-1700 Fribourg, Switzerland

 Record created 2006-03-10, last modified 2018-03-17

Rate this document:

Rate this document:
(Not yet reviewed)