Introduction to Optax and Optimizers

Almost whole Machine Learning centers around optimization. For example, we can apply optimization to find the parameters corresponding to the minimization of both convexTechnically, a function that doesn’t go above the line connecting any two points on the function. Convex functions are very useful as their optimization is straightforward. and non-convexA function that is not convex. Unfortunately, the complexity of neural networks leads to having non-convex objective functions. Optimizing them is usually done with some approximate approaches. loss functions.

This lesson will review the basic theory and syntax of Optax. This information will be consolidated in the next lesson with a review of some advanced functions and a working example.

Common loss functions

Before we begin, take a look at the subtle difference between loss and error functions: namely, a loss function is defined over the individual samples while an error function is defined over a whole batch or dataset.

Now we’ll review some common loss functions.

$L_2$

L2 loss simply takes the $L_2$ norm (or Euclidean) distance between the intended and original outputs.

$$\ell_{i(L_2)} = \frac {1}{2}\sqrt{y_i^{'2}-y_i^2}$$ ...