What are tree recursive neural networks (Tree-RNNs)?
Overview
A Recursive Neural Network (RNN) falls under the category of a deep neural network. It is constructed in such a way that it includes applying the same set of weights
A simple RNN structure is illustrated in the figure below:
A recursive neural network uses the learning algorithms, which means it predicts the output values (
Implementation
Tree RNNs are commonly used to implement the score which is calculated at each traversal of nodes that tells us which pair of phrases and words need to be combined first to form the best syntactic tree which explains a given sentence. The question arises, how do we actually calculate the score and merge the phrases/inputs? Here's an architecture of simple Tree-RNN through which we will explain the working of a recursive neural network.
In our illustration, x1 and x2 are carrying the words. At the hidden layer, these words are merging in the form of vector representation p, which is calculated through the following recurrence relation.
Where,
f = Non-linear activation function ( Sigmoid, Tanh, and so on).
W[x1;x2]: Concatenation of x1 and x2 which is then multiplied by weight W.
b = Bias
After the merged pair calculation, we need to find the score S at each node which is calculated by the following relation:
We will get S by multiplying particular weight W and merging vector P. This enables us to find the best pair of
Example
The above illustration shows that there are different words at each input, and they are combined together at the hidden layer. This procedure is done by calculating P and S. The traversing is done at each node and the score is calculated recursively. In the end, all the words are merged at the top and form a sentence which in our example is "A small crowd quietly enters the historic church."
Code
# initialize the recursive neural network and the hidden layersmy_rnn = RNN()hidden_layers = [0, 0, 0, 0, 0, 0, 0, 0]sequence = ["A", "small", "crowd", "quietly","enters","the","historic","church"]# feeding the sequence as input to the networkfor word in sequence:prediction, hidden_layers = my_rnn(word, hidden_layers)# prediction for the next wordnext_word = prediction
Explanation
Lines 2 and 3: We initialize the recursive neural network and hidden layers.
Line 5: We enter sample input data in the form of verbs/nouns/phrases.
Lines 8 and 9: We loop through the given input data sequence which feeds both the current word and the previous internal state into the RNN. A structured prediction is then made and the internal state is updated.
Line 12: We return the output from the variable-size input data.
Conclusion
Therefore, a recursive neural network is a tree-based structure that uses the learning algorithms for
