Appendix B: GANs Learn Likelihood

What does a GAN actually learn? This is a very good question which doesn’t have an immediately obvious answer.

Here we’ll develop an intuition for what a GAN learns in a way that avoids lots of mathematical jargon. Let’s start with what a GAN doesn’t learn.

GANs don’t memorize training data

A GAN does not learn to memorize examples from the training data. A GAN doesn’t even learn to memorize small parts of the training data examples. For training data consisting of faces, this means the generator doesn’t memorise elements like eyes, ears, lips, or noses.

The generator doesn’t see the training data directly. All it can learn from is the back-propagated error feedback from a discriminator which itself can only make a binary conclusion that an image is real or fake.

GANs instead learn the likelihood of elements appearing in the training data.

Simplified example

The following picture shows a very tiny dataset of just 8 images. The images are also tiny, 3 pixels by 3 pixels in size. Furthermore, the pixels can only be one of two values, which we’ll show here as blue and white.