...

/

Appendix B: GANs Learn Likelihood

Appendix B: GANs Learn Likelihood

Get to know how GANs learn likelihood.

What does a GAN actually learn? This is a very good question which doesn’t have an immediately obvious answer.

Here we’ll develop an intuition for what a GAN learns in a way that avoids lots of mathematical jargon. Let’s start with what a GAN doesn’t learn.

GANs don’t memorize training data

A GAN does not learn to memorize examples from the training data. A GAN doesn’t even learn to memorize small parts of the training data examples. For training data consisting of faces, this means the generator doesn’t memorise elements like eyes, ears, lips, or noses.

The generator doesn’t see the training data directly. All it can learn from is the back-propagated error feedback from a discriminator which itself can only make a binary conclusion that an image is real or fake.

GANs instead learn the likelihood of elements appearing in the training data.

Simplified example

The following picture shows a very tiny dataset of just 8 images. The images are also tiny, 3 pixels by 3 pixels in size. Furthermore, the pixels can only be one of two values, which we’ll show here as blue and white.

If we, as humans, were asked to draw images that looked like they could have come from this dataset, we would intuitively draw them with a blue pixel in the middle and top left. We might draw some that had the bottom left pixel coloured blue too.

Behind that intuition is an understanding of how likely a pixel is blue. We can see that most images have a blue middle pixel. In fact, all of them except one do. Many also have a blue top-left pixel. A few have the bottom left pixel set to blue. Some pixels, like the top middle one, are never blue.

Let’s move from intuition to actual calculation. We can ...