Nesterov Accelerated Gradient (NAG)

What is NAG?

Consider the scenario where a company wants to determine the optimal production rate and the optimal selling price for one of its products to maximize profit, which is given by a non-convex objective having several local optimums.

NAG is a variant of gradient descent with momentum that improves the convergence rate and the stability of gradient descent. The main idea is to use a look-ahead term to calculate the gradient at a future point rather than the current point. This way, the algorithm can anticipate the direction of the optimal solution and avoid overshooting or oscillating. The figure below illustrates the idea:

The NAG update at a time $t$ is given as follows:

Here, $v_t$ is the velocity vector that accumulates the gradient over time, $\beta$ is the momentum coefficient that controls how much of the previous velocity is retained, $\alpha$ is the learning rate that scales the gradient, $\nabla_\theta J$ ...