What is Xavier/Glorot initialisation?
In the field of deep learning, we use the Xavier method to initialize the weights of neural networks to mitigate the problem of vanishing gradients and exploding gradients. Xavier Glorot introduced this method in 2010. The main purpose of initializing weights through the Xavier method is to propagate effectively during forward and backward propagation.
In this Answer, we will discuss the mathematical intuition behind the Xavier/Glorot technique and also learn to initialize Xavier weights in Python through code examples.
Mathematical intuition
The mathematical intuition behind Xavier's method is given below:
Let
, be a fully connected layer of a neural network. Let
be the input and output units of this layer respectively. The dimension of the weight matrix
of an is represented with . In the Xavier method, we want to achieve the balance initialization of the weights. The variance of the weights in Xavier initialization is inversely proportional to the sum of the input n and output m units.
The weights in the Xavier method are initialized with a mean at 0 and the standard deviation is the square root of the above-calculated variance.
Code
The code shown below illustrates how to initialize the Xavier weights in Python.
import numpy as npdef xavier_initializer(input_units, output_units):variance = 1 / (input_units + output_units)std_dev = np.sqrt(variance)weights = np.random.normal(loc=0.0, scale=std_dev, size=(input_units, output_units))return weights# The shape of the weight tensorinput_units = 100output_units = 200# Weights with Xavier initializationweights = xavier_initializer(input_units, output_units)
The explanation of this code is as follows:
Line 6: Standard deviation is calculated as explained above.
Line 7: Weights are initialized with mean 0 and standard deviation. The loc here represents the mean. The size of weights is according to the input and output units.
Benefits
There are several benefits of Xavier initialization, some of which are mentioned below:
By initializing the weights properly, we can ensure that there is a balanced flow of information during forward and backward propagation.
It enables the neural network to train in an efficient way.
It reduces the likelihood to get stuck at local minima.
It trains the neural network to learn complex patterns in the dataset.
Limitations
There is a limitation of Xavier initialization. It was primarily introduced for nonlinear activation functions such as sigmoid and tan etc. For activation functions such as ReLU and its variants, we use the He initialization.
Conclusion
In this Answer, we discussed the mathematical intuition, code, benefits, and limitations of using Xavier initialization. To check how much you learned, let's dive into a quiz.
Which of the following statements about the Xavier/Glorot initialization is correct?
Xavier initialization sets all the weights in a neural network to the same value.
Xavier initialization is only suitable for networks with linear activation functions.
Xavier initialization is based on the assumption that the inputs and outputs have different variances.
Xavier initialization helps mitigate the issue of vanishing or exploding gradients during training.
Free Resources
