What are recurrent neural networks?
A recurrent neural network (RNN) is a class of artificial neural networks that contains loops within its hidden layers, and allows information to
Unlike traditional neural networks, which utilize a simple feed-forward flow, recurrent neural networks contain a looping mechanism that computes an internal state update with each time step. This gives an RNN the capability to retain information about preceding elements in a series of data. This is particularly useful because it allows the RNN to handle varied lengths of data, maintain the order within a sequence of data, and share parameters across neurons in the network. Recurrent neural networks operate based on a recurrence relation that is given by:
We can see from the recurrence relation that the internal state at a given time step is dependent on the internal state at the previous time step and the current input.
For example, if we take the activation function to be tanh, then the internal state at time t can be computed as tanh applied to the summation of
The output at the time t would therefore be a multiplication of the internal state at that time by the separate weight.
Recurrent neural networks are mainly used to analyze
- Speech recognition
- Natural language processing
- Video captioning
- Text processing
- Language Translation
We can see from the examples of applications above that recurrent neural networks are particularly useful for addressing issues where the order of the input data is of great importance.
# initialize the neural network and the hidden layersmy_rnn = RNN()hidden_layers = [0, 0, 0, 0]sequence = ["I", "love", "recurrent", "neural"] #example data sequence# feeding the sequence as input to the networkfor word in sequence:prediction, hidden_layers = my_rnn(word, hidden_layers)# prediction for the next wordnext_word = prediction
-
Lines 2 and 3 initialize the RNN and the hidden layers within the network.
-
Line 6 sets up the input data.
-
Lines 9 and 10 loop through the sequence and feed both the current word and the previous internal state into the RNN. A prediction is then made and the internal state is updated.
-
Line 13 returns the output of the RNN, which is the predicted word.
