...

/

Creating a DBN with the Keras Model API

Creating a DBN with the Keras Model API

Learn how to implement deep belief networks (DBN) using TensorFlow 2.0.

The creation of a single-layer RBM to generate images is the building block required to create a full-fledged DBN. Usually, for a model in TensorFlow 2, we only need to extend tf.keras.Model and define an initialization (where the layers are defined) and a call function (for the forward pass). For our DBN model, we also need a few more custom functions to define its behavior.

Implementing the RBM

Let’s continue by combining multiple RBMs in layers to create a more powerful model—the deep belief network (DBN).

RBM initialization

First, in the initialization, we need to pass a list of dictionaries that contain the parameters for our RBM layers (number_hidden_unitsnumber_visible_units, and learning_rate, cd_steps):

Press + to interact
class DBN(tf.keras.Model):
def __init__(self, rbm_params=None, name='deep_belief_network', num_epochs=100,
tolerance=1e-3, batch_size=32, shuffle_buffer=1024, **kwargs):
super().__init__(name=name, **kwargs)
self._rbm_params = rbm_params
self._rbm_layers = list()
self._dense_layers = list()
for num, rbm_param in enumerate(rbm_params):
self._rbm_layers.append(RBM(**rbm_param))
self._rbm_layers[-1].build([rbm_param["number_visible_units"]])
if num < len(rbm_params)-1:
self._dense_layers.append(
tf.keras.layers.Dense(rbm_param["number_hidden_units"],
activation=tf.nn.sigmoid))
else:
self._dense_layers.append(
tf.keras.layers.Dense(rbm_param["number_hidden_units"],
activation=tf.nn.softmax))
self._dense_layers[-1].build([rbm_param["number_visible_units"]])
self._num_epochs = num_epochs
self._tolerance = tolerance
self._batch_size = batch_size
self._shuffle_buffer = shuffle_buffer

At the same time, we also initialize a set of sigmoidal dense layers with a softmax at the end, which we can use for fine-tuning through backpropagation once we’ve trained the model using the generative procedures outlined earlier.

To train the DBN, we begin a new code block to start the generative learning process for the stack of RBMs:

Press + to interact
# pretraining:
inputs_layers = []
for num in range(len(self._rbm_layers)):
if num == 0:
inputs_layers.append(inputs)
self._rbm_layers[num] = self.train_rbm(self._rbm_layers[num], inputs)
else:
# Pass all data through the previous layer
inputs_layers.append(inputs_layers[num-1].map(
self._rbm_layers[num-1].forward))
self._rbm_layers[num] = self.train_rbm(
self._rbm_layers[num], inputs_layers[num])

It initializes an empty list inputs_layers ...