Higher-Order Gradients: Hessian

Higher-order derivatives

Gradients and derivatives are also known as first-order derivatives. Algorithms like Newton’s method require computing higher-order derivatives, such as second-order derivatives. Given a multivariate function $f(x_1, x_2, ..., x_m): \R^m \rightarrow \R$, higher-order gradients/derivatives are represented as follows:

• $\frac{\partial^2 f}{\partial x_i^2}$ is known as the second partial derivative of $f$ with respect to $x_i$. In other words, the second-order derivative is the derivative of the first-order derivative $\frac{\partial}{\partial x_i} \frac{\partial f}{\partial x_i}$ .

• $\frac{\partial^n f}{\partial x_i^n}$ ...