Hands-On Generative Adversarial Networks with PyTorch/

...

/

Defining Activation Functions and The Loss Function

Defining Activation Functions and The Loss Function

Understand the concepts of activation functions and loss function and practice generating samples using a basic GAN.

We'll cover the following...

We will be only using NumPy to calculate and train our GAN model (and optionally using Matplotlib to visualize the signals). All of the following code can be placed in a simple .py file (such as simple_gan.py). Let’s look at the code step by step:

  1. Import the numpy library:

Press + to interact
import numpy as np

  1. Define a few constant variables that are needed in our model:

Press + to interact
Z_DIM = 1
G_HIDDEN = 10
X_DIM = 10
D_HIDDEN = 10
step_size_G = 0.01
step_size_D = 0.01
ITER_NUM = 50000
GRADIENT_CLIP = 0.2
WEIGHT_CLIP = 0.25

  1. Define the real sine samples (with numpy.sin) that we want to estimate:

Press + to interact
def get_samples(random=True):
    if random:
        x0 = np.random.uniform(0, 1)
        freq = np.random.uniform(1.2, 1.5)
        mult = np.random.uniform(0.5, 0.8)
    else:
        x0 = 0
        freq = 0.2
        mult = 1
    signal = [mult * np.sin(x0+freq*i) for i in range(X_DIM)]
    return np.array(signal)

In the previous snippet, we use a bool variable named random to introduce randomness into the real samples, as real-life data has. The real samples look like this (50 samples with random=True):

Press + to interact
The real sine samples
The real sine samples

  1. Define the activation functions and their derivatives. If you're not familiar with the concept of activation functions, just remember that their jobs are to adjust the outputs of a layer so that its next layer can have a better understanding of these output values:

Press + to interact
def ReLU(x):
    return np.maximum(x, 0.)
def dReLU(x):
    return ReLU(x)
def LeakyReLU(x, k=0.2):
    return np.where(x >= 0, x, x * k)
def dLeakyReLU(x, k=0.2):
    return np.where(x >= 0, 1., k)
def Tanh(x):
    return np.tanh(x)
def dTanh(x):
    return 1. - Tanh(x)**2
def Sigmoid(x):
    return 1. / (1. + np.exp(-x))
def dSigmoid(x):
    return Sigmoid(x) * (1. - Sigmoid(x))

  1. Define a helper function to initialize the layer parameters:

Press + to interact
def weight_initializer(in_channels, out_channels):
    scale = np.sqrt(2. / (in_channels + out_channels))
    return np.random.uniform(-scale, scale, (in_channels,
    out_channels))

  1. Define the loss function (both forward and backward):

Press + to interact
class LossFunc(object):
    def __init__(self):
        self.logit = None
        self.label = None
    def forward(self, logit, label):
        if logit[0, 0] < 1e-7:
            logit[0, 0] = 1e-7
        if 1. - logit[0, 0] < 1e-7:
            logit[0, 0] = 1. - 1e-7
        self.logit = logit
        self.label = label
        return - (label * np.log(logit) + (1-label) * np.log(1- logit))
    def backward(self):
        return (1-self.label) / (1-self.logit) - self.label/self.logit

This is called binary cross-entropy, which is typically used in binary classification problems (in which a sample either belongs to class A or class B). Sometimes, one of the networks is trained too well so that the sigmoid output of the discriminator might be either too close to 0 or 1. Both of the scenarios lead to numerical errors of the log function. Therefore, we need to restrain the maximum and minimum values of the output value.

Working on forward pass and backpropagation

Now, let's create our generator and discriminator networks. We put the ...