Forward and Backward Propagation in RNNs

Forward propagation

This is the process of calculating and storing the intermediate variables (including outputs) for the neural network in order from the input layer to the output layer. We will not discuss this in too much detail, as it is assumed you already have some background knowledge on this.

A simple architecture of an RNN is shown below:

Explanation:

• $a^{0}$ denotes the initial state of the network.
• $a^{M}$, where M varies from 1 to N (number of cells), denotes the state of the previous cell’s output. This also denotes the information that has been read so far.
• $x^{M}$, where M varies from 1 to N (number of input words), denotes the $M^{th}$ input at the $M^{th}$ time step.
• $Y^{M}$, where M varies from 1 to N (number of cells), denotes the output of the $M^{th}$ RNN cell.

To get a better understanding, let’s look at a single RNN cell as shown below.

Explanation:

• $a^{t}$

...