Higher-Order Gradients: Hessian

Higher-order derivatives

Gradients and derivatives are also known as first-order derivatives. Algorithms like Newton’s method require computing higher-order derivatives, such as second-order derivatives. Given a multivariate function $f(x_1, x_2, ..., x_m): \R^m \rightarrow \R$, higher-order gradients/derivatives are represented as follows:

• $\frac{\partial^2 f}{\partial x_i^2}$ is known as the second partial derivative of $f$ with respect to $x_i$. In other words, the second-order derivative is the derivative of the first-order derivative $\frac{\partial}{\partial x_i} \frac{\partial f}{\partial x_i}$ .

• $\frac{\partial^n f}{\partial x_i^n}$ is the $n$th order derivative of $f$ with respect to $x_i$.

• $\frac{\partial^2 f}{\partial x_j \partial x_i} = \frac{\partial}{\partial x_j} \frac{\partial f}{\partial x_i}$ is the partial derivative obtained by first partial differentiating with respect to $x_i$ and then with respect to $x_j$.

• $\frac{\partial^2 f}{\partial x_i \partial x_j} = \frac{\partial}{\partial x_i} \frac{\partial f}{\partial x_j}$ is the partial derivative obtained by first partial differentiating with respect to $x_j$ and then with respect to $x_i$.

The Hessian matrix $H$ is a collection of all the second-order partial derivatives with the $(i,j)^{th}$ entry given as $H_{ij} = \frac{\partial^2 f}{\partial x_i \partial x_j}$. In other words,