Experimentation

Run your own experiments

If you’ve worked through to this stage you’ve covered enough basics to meaningfully try your own ideas for improving the GAN process.

• You could try different kinds of loss functions, different sizes of the neural network, perhaps even variations of the standard GAN training loop.
• Perhaps you might try to discourage mode collapse by including a measure of diversity over several outputs in the loss function.
• If you’re very confident, you might try to implement your own optimiser, one which is better suited to the adversarial dynamics of a GAN.

Experiment with GELU activation function

The following is one of my own simple experiments with an activation function called GELU, which is like a ReLU but has a softer corner.

📝 Some have suggested that such activation functions are now the state of the art because they provide good gradients, and don’t have a sharp discontinuity around the origin.