Introduction to Optax and Optimizers

Almost whole Machine Learning centers around optimization. For example, we can apply optimization to find the parameters corresponding to the minimization of both convexTechnically, a function that doesn’t go above the line connecting any two points on the function. Convex functions are very useful as their optimization is straightforward. and non-convexA function that is not convex. Unfortunately, the complexity of neural networks leads to having non-convex objective functions. Optimizing them is usually done with some approximate approaches. loss functions.

This lesson will review the basic theory and syntax of Optax. This information will be consolidated in the next lesson with a review of some advanced functions and a working example.