Understanding RNNs

In this lesson, we’ll discuss what an RNN is by starting with a gentle introduction and then moving on to more in-depth technical details. We mentioned earlier that RNNs maintain a state variable that evolves over time as the RNN sees more data, giving it the power to model sequential data. In particular, this state variable is updated over time by a set of recurrent connections. The existence of recurrent connections is the main structural difference between an RNN and a feed-forward network. The recurrent connections can be understood as links between a series of memories that the RNN learned in the past, connecting to the current state variable of the RNN. In other words, the recurrent connections update the current state variable with respect to the past memory the RNN has, enabling the RNN to make a prediction based on the current input as well as the previous inputs.

Note: The term “RNN” is sometimes used to refer to the family of recurrent models, which has many different models. In other words, it is sometimes used as a generalization of a specific RNN variant. Here, we are using the term RNN to refer to one of the earliest implementations of an RNN model known as the Elman network.

In the upcoming lessons, we'll discuss the following topics. First, we’ll discuss how we can start by representing a feed-forward network as a computational graph. Then, we’ll see through an example why a feed-forward network might fail at a sequential task. Then, we’ll adapt that feed-forward graph to model sequential data, which will give us the basic computational graph of an RNN. We’ll also discuss the technical details (for example, update rules) of an RNN. Finally, we’ll discuss the details of how we can train RNN models.

The problem with feed-forward neural networks

To understand the limits of feed-forward neural networks and how RNNs address them, let’s imagine a sequence of data: