Generating New Text with the Model

Defining the inference model

During training, we trained our model and evaluated it on sequences of bigrams. This works for us because during training and evaluation, we have the full text available to us. However, when we need to generate new text, we don’t have anything available to us. Therefore, we have to make adjustments to our trained model so that it can generate text from scratch.

The way we do this is by defining a recursive model that takes the current time step’s output of the model as the input to the next time step. This way, we can keep predicting words or bigrams for an infinite number of steps. We provide the initial seed as a random word or bigram picked from the corpus (or even a sequence of bigrams).

The figure below illustrates how the inference model works.