Creating Separable Encodings of Images

Let’s examine how neural networks can optimally compress information into numerical vectors. To do so, each element of the vector should encode distinct information from the others, a property we can achieve using a variational objective. This variational objective is the building block for creating VAE networks.

In the figure below, you can see an example of images from the CIFAR-10 datasetKrizhevsky, Alex. 2009. “CIFAR-10 and CIFAR-100 Datasets.” Toronto.edu. 2009. https://www.cs.toronto.edu/~kriz/cifar.html., along with an example of an early VAE algorithm that can generate fuzzy versions of these images based on a random number inputPiyush Malhotra, “Variational Autoencoders”, Autoencoder-Implementations. GitHub repository. 2018, https://www.piyushmalhotra.in/Autoencoder-Implementations/VAE/. More recent work on VAE networks has allowed these models to generate much better images.

The Restricted Boltzmann Model (or deep belief network) model involves learning the posterior probability distribution for images ($x$) given some latent code ($z$), represented by the hidden layer(s) of the network, the marginal likelihood ofKingma, Diederik, and Max Welling. 2014. “Auto-Encoding Variational Bayes.” https://arxiv.org/pdf/1312.6114.pdf.:

We can see $z$ as being an “encoding” of the image $x$ (for example, the activations of the binary hidden units in the RBM), which can be decoded (for example, run ...