Auto-differentiation

Background

First pioneered by the seminal work of Rumelhart and Hinton in 1986, the majority of current machine learning optimization methods use derivatives. So, there is a pressing need for their efficient calculation.

Manual calculations

Most of the early machine learning researchers and scientists $-$ for example, Bottou, 1998 for Stochastic Gradient Descent $-$ had to go through a slow, laborious process of manual calculation of analytical derivatives, which is prone to error.

Using computer program

Programming-based solutions are less laborious, but calculating these derivatives in a program can also be tricky. We can categorize them into three paradigms:

1. Symbolic differentiation
2. Numeric differentiation
3. Auto differentiation

The first and second methods are prone to errors, including:

• Calculating higher-derivatives is tricky due to long and complex expressions for symbolic differentiation and rounding-off errors $—$ meaning less accurate results $—$ in numeric differentiation.
• Numeric differentiation uses discretization, which results in loss of accuracy.
• Symbolic differentiation can lead to inefficient code.
• Both are slow to calculate the partial derivatives, a key feature of

...