In this chapter, we will work on GANs that directly generate sequential data, such as text and audio. While doing so, we will go through image-synthesizing models so that we can become familiar with NLP models quickly.

Teaching GANs how to tell jokes

When it comes to the generation of text, the biggest difference in terms of image generation is that text data is discrete while image pixel values are more continuous, though digital images and text are both essentially discrete. A pixel typically has 256 values, and slight changes in the pixels won’t necessarily affect the image’s meaning to us. However, a slight change in the sentence, even a single letter (for example, turning we into he), may change the whole meaning of the sentence. Also, we tend to have a higher tolerance bar for synthesized images compared to text. For example, if 90% of the pixels in the generated image of a dog are nearly perfect, we may have little trouble recognizing the dog because our brains are smart enough to automatically fill in the missing pixels. However, if we are reading a piece of news in which every one out of 10 words doesn’t make any sense, we will definitely find it hard to enjoy reading it. This is why text generation is hard, and there’s less remarkable progress in text generation than in image synthesis.

SeqGANYu, Lantao, Weinan Zhang, Jun Wang, and Yong Yu. "Seqgan: Sequence generative adversarial nets with policy gradient." In Proceedings of the AAAI conference on artificial intelligence, vol. 31, no. 1. 2017. was one of the first successful attempts at text generation with adversarial learning. In this chapter, we will walk through the design of SeqGAN, how to create our own vocabulary for NLP tasks, and how to train SeqGAN so that it can generate short jokes.

Get hands-on with 1400+ tech skills courses.