What is forward propagation in neural networks?

There are two essential steps to optimize a neural network. These steps are as follows:

1. Forward propagation: This is a technique used to find the actual output of neural networks. In this step, the input is fed to the network in a forward direction. It helps us find the actual output of each neuron.

2. Backpropagation: In this step, we update the weights of the network based on the difference between the actual output of the network and the ground truth.

In this answer, we'll focus more on forward propagation.

To learn about backpropagation click here.

Computation

There are two main steps in forward propagation—calculating the sum of the product and using an activation function.

The sum of the product

For each neuron, we multiply the inputs $[x_0, x_1, .., x_n]$ with their respective weights $[w_0, w_1, ... , w_n]$ such that $x_0$ is multiplied with $w_0$, $x_1$ is multiplied with $w_1$, and so on. We then add all the products and add a bias $b$ in it.

In the following illustration, we depict the computation process above for a neuron over three inputs and a bias.

Then sum of products$S$will pass through an activation function to compute the actual output of a neuron.

Activation function

The activation function converts the neuron’s output into a particular output range and introduces non-linearity. It converts it in the range of $0$ to $1$ or $-1$ to $1$, depending on the activation function used.

To learn more about activation functions click here.

The following illustration shows the forward propagation of a neural network with an input layer, a hidden layer consisting of two neurons, and an output layer:

We are assuming a linear activation function in the example above. A linear function gives the same output as its input: