Embedding Matrix
Create a trainable embedding matrix to calculate word embeddings.
We'll cover the following...
Chapter Goals:
- Learn how the embedding matrix is initialized during training
- Create a function for the forward run of embedding training
A. Variable creation
In TensorFlow, we use the tf.layers.dense function to create each layer of a fully-connected neural network. This is a function provided by TensorFlow that automatically handles all the setup required for the neural network's weights. So previously all we needed to do was initialize the weight variables (via tf.global_variables_initializer) prior to running the computation graph.
However, for our embedding model, the variable we need to create is a 2-D matrix that contains the embedding vectors for each vocabulary word ID. Therefore, we need to manually create this variable using the tf.compat.v1.get_variable function.
The function only takes in one required argument, which is the name of the variable. Of the possible keyword arguments, a couple important ones to know are shape and dtype, which allow us to ...