A Simplified Natural Gradient Learning Algorithm

Abstract EN

Adaptive natural gradient learning avoids singularities in the parameter space of multilayer perceptrons.

However, it requires a larger number of additional parameters than ordinary backpropagation in the form of the Fisher information matrix.

This paper describes a new approach to natural gradient learning that uses a smaller Fisher information matrix.

It also uses a prior distribution on the neural network parameters and an annealed learning rate.

While this new approach is computationally simpler, its performance is comparable to that of adaptive natural gradient learning.