Search⌘ K
AI Features
Home
Courses
Deep Learning with PyTorch Step-by-Step: Part I - Fundamentals

Mini-Batch Inner Loop and Training Split

Explore how to structure the mini-batch inner loop in PyTorch, implementing efficient gradient descent by processing data in smaller batches. Understand how to use PyTorch's random_split function to create training and validation datasets, and build corresponding data loaders. This lesson helps you develop a lean and effective training loop while managing data splits for model evaluation.

We'll cover the following...

The inner loop

From now on, it is very unlikely that you will ever use a (full) batch gradient descent again, both in this course or in real life. So once again, it makes sense to organize a piece of code that is going to be used repeatedly into its own function: the mini-batch inner loop!

The inner loop depends on three elements:

  • The device where data is being sent to.

  • A data loader to draw mini-batches from.

  • A step function, returning the corresponding loss.

Taking these elements as inputs and using them to perform the inner loop, we will end up with a function like this:

Python 3.5
def mini_batch(device, data_loader, step):
    mini_batch_losses = []
    for x_batch, y_batch in data_loader:
        x_batch = x_batch.to(device)
        y_batch = y_batch.to(device)
        mini_batch_loss = step(x_batch, y_batch)
        mini_batch_losses.append(mini_batch_loss)
    loss = np.mean(mini_batch_losses)
    return loss

In the last lesson, we realized that we were executing five times more updates (the train_step function) per epoch due to the mini-batch inner loop. Before, 1,000 epochs meant 1,000 ...