Summary: Teaching Networks to Generate Digits

In this chapter, we learned about one of the most important models from the beginnings of the deep learning revolution, the DBN. We saw that DBNs are constructed by stacking together RBMs, and how these undirected models can be trained using CD.

The chapter then described a greedy, layer-wise procedure for priming a DBN by sequentially training each of a stack of RBMs, which can then be fine-tuned using the wake-sleep algorithm or backpropagation. We then explored practical examples of using the TensorFlow 2 API to create an RBM layer and a DBN model, illustrating the use of the GradientTape class to compute updates using CD.

Then, we explored the wake-sleep algorithm and learned how we can compile the DBN as a normal deep neural network and perform backpropagation for supervised training. We applied these models to MNIST data and learned how an RBM can generate digits after training converges and has features resembling the convolutional filters described earlier.