Minibatch Gradient Descent

Stochastic gradient descent (SGD)

Recall that to compute the gradient $\nabla_\theta J(\theta)$ of an objective $J(\theta)$, we need to aggregate the gradients $\nabla_\theta \mathcal{L}(f_\theta(x_i), y_i)$ for the whole dataset $D = \{ (x_i, y_i) \}_{i=1}^N$ to perform just one update. This makes vanilla gradient descent very slow and intractable for large datasets, such as ImageNet having one million images, that do not fit in memory.

SGD, in contrast, performs the update based on one example at a time as follows: