What is the difference between epoch, batch size, and iteration?

Epoch, batch size, and iteration are some machine learning terminologies that one should understand before diving into machine learning. We understand these terms one by one in the following sections.

Epoch

An epoch completes once a whole dataset has undergone forward propagation and backpropagation.

Passing the whole dataset at once causes the underfitting of the curve. Therefore, we must give the entire dataset through the neural network model more than once to make the fitting curve from underfitting to optimal. But it can also cause overfitting of the curve if there are more epochs than needed.

Batch size

The total number of data points in a single batch passed through the neural networks is called batch size.

An epoch is made up of batches. Sometimes the whole dataset can not be passed through the neural network at once due to insufficient memory or the dataset being too large. We divide the entire dataset into smaller numbers of parts called batches. These batches are passed through the model for training.

Note: The number of batches and batch size are different.

Iteration

The total number of batches needed to complete one epoch is called iteration. For example, the dataset consists of 1000 images. We divide it into ten batches of size 100 each. One iteration is completed when one batch passes through a neural network that is forward propagation and backpropagation.

The following equation shows the relation between batch size and iteration with epoch and can be used to calculate epoch size:

The following illustration can help in understanding the difference between the above terminologies: