Deep Learning with PyTorch Step-by-Step: Part I - Fundamentals/

...

/

Mini-Batch Inner Loop and Training Split

Mini-Batch Inner Loop and Training Split

Learn about how you can implement the inner loop of mini-batch, add it into your training loop, and get a glimpse at using "random_split".

We'll cover the following...

The inner loop

From now on, it is very unlikely that you will ever use a (full) batch gradient descent again, both in this course or in real life. So once again, it makes sense to organize a piece of code that is going to be used repeatedly into its own function: the mini-batch inner loop!

The inner loop depends on three elements:

  • The device where data is being sent to.

  • A data loader to draw mini-batches from.

  • A step function, returning the corresponding loss.

Taking these elements as inputs and using them to perform the inner loop, we will end up with a function like this:

Press + to interact
def mini_batch(device, data_loader, step):
    mini_batch_losses = []
    for x_batch, y_batch in data_loader:
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        mini_batch_loss = step(x_batch, y_batch)
        mini_batch_losses.append(mini_batch_loss)
    loss = np.mean(mini_batch_losses)
    return loss

In the last lesson, we realized that we were executing five times more updates (the train_step function) per epoch due to the mini-batch inner loop. Before, 1,000 epochs ...