Coordinate Descent

The coordinate descent algorithm

Consider a multivariate function $f(x) = f(x_1, x_2, ..., x_n)$ that we want to optimize. Using the gradient descent algorithm, the updates will happen in all directions at once.

Coordinate descent is a variation of gradient descent that tries to find the minimum of a function by minimizing it (i.e., performing gradient descent) along one coordinate direction at a time. Starting from an initial point $x^0$, coordinate descent defines $x^t$ at the time $t>0$ by solving $n$ single variable optimization problems one by one as follows: