Learning Rate

Introduction to the learning rate

The learning rate is the most important hyper-parameter. There is a huge amount of material on how to choose a learning rate, how to modify the learning rate during the training, and how the wrong learning rate can completely ruin the model training.

Maybe you might have seen this famous graph (from Stanford’s CS231n class) that shows how a learning rate that is too big or too small affects the loss during training. This is pretty much general knowledge, but it needs to be thoroughly explained and visually demonstrated to be truly understood. So, let us start!

To start it off, I will tell you a little story (trying to build an analogy here; please bear with me).