Learn to solve optimization problems in Python using essential math tools, metaheuristic methods, and constrained optimization techniques. Master efficient solutions for real-world applications.

Optimization with Python from zero to hero.png

cvxpy.tar.gz

cvxpy

Optimization theory seeks the best solution, which is pivotal for machine learning, cost-cutting in manufacturing, refining logistics, and boosting finance profits.

This course provides a detailed description of different optimization problems and the techniques used to solve them. You’ll begin with the formal definition of an optimization problem and an overview of essential mathematical tools: derivatives, gradients, and Hessian. With this knowledge, you’ll implement solutions for several optimization problems. You’ll also learn approximate metaheuristic population methods like particle swarm optimization and genetic algorithms, and solve constrained optimization problems using techniques like penalty methods and constraint simplification. Finally, you’ll solve linear programs and integer programs. 

By the end of this course, you’ll become proficient in formulating different kinds of problems as optimization problems and become skilled in the Python tools that efficiently solve these problems.

Mastering Optimization with Python

So far, we've seen how we can draw a function with a single variable. The graph of the function is a curve that follows a particular pattern depending on the formula that describes the function.

This possibility of drawing the function has been very useful. It helped us solve some simple problems and understand the concept of derivatives. We strongly recommend looking at the graph of a function whenever it's possible. It's always a good start.

But unfortunately, it's not always possible to draw the graph of a function. For example, if the function has two variables instead of one, then the graph will be in 3D. The graph of the function is not a line anymore, it's a region with height, width, and depth.

We can still draw 3D graphs. But what happens when functions have more than two variables? How can we draw a 4D or 5D graph? It's impossible!



Derivatives will help us again to solve these problems. Let's see how to use derivatives in multiple dimensions.

# Derivatives in multiple dimensions

Remember when we calculated derivatives, we used a line like this one:

```python
df = diff(f, x)
``` 

This means we're calculating the derivative of ```f``` with respect to ```x```. When we have just one variable, we always calculate the derivative of the function with respect to the same variable.

In the same way, we can calculate the derivative of a function of multiple variables with respect to just one of its variables. When we do that, we consider the rest of the variables as constant numbers.

Let's see an example that calculates the derivatives of the function $f(x, y, z) = x^2 + y^2 + z^2$.


from sympy import symbols, diff

# Defining the symbols we'll use
x, y, z = symbols('x y z')

# We can play with this function later
f = x**2 + y**2 + z**2

df_x = diff(f, x)
print(f"Derivative w.r.t. x:", df_x)

df_y = diff(f, y)
print(f"Derivative w.r.t. y:", df_y)

df_z = diff(f, z)
print(f"Derivative w.r.t. z:", df_z)

This is what the previous code does, line by line:

* **Line 1:** We import the ```symbol``` and ```diff``` functions from SymPy.
* **Line 4:** We define the symbols we're going to use. In this case, we'll use three variables: $x$, $y$, and $z$.
* **Line 7:** Here we define the function $f(x, y, z) = x^2 + y^2 + z^2$.
* **Lines 9 and 10:** We calculate and print the derivative of $f$ with respect to $x$. We pass $f$ and $x$ as arguments to SymPy's ```diff``` function.
* **Lines 12 and 13:** Similar to the previous lines, but here we take the derivative with respect to $y$.
* **Lines 15 and 16:** Similar to the previous lines, but here we take the derivative with respect to $z$.

Now that we know how to calculate derivatives in multiple dimensions using Python, let's visit another important concept that we'll be using in this and the next section of the course: the gradient.

>**Note:** See how every derivative is a new function. In calculus, these derivatives we calculate for one variable are called **partial derivatives**.

# The  gradient

As we saw, a function has a derivative for each variable in its formula. In mathematical notation, we write $\frac{\partial f}{\partial x}$ to refer to the derivative of $f$ with respect to $x$. In the same way, we write $\frac{\partial f}{\partial y}$, $\frac{\partial f}{\partial z}$, etc., to refer to the rest of the derivatives.

The **gradient** is the vector of all those derivatives. We write it like $(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z}, ...)$.

It's the sorted disposition of each partial derivative, and it's denoted by $\nabla f$.

$$
\nabla f = (\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y}, \frac{\partial f}{\partial z}, ...)
$$

For example, for the function $f = x^2 + y^2 + z^2$, we have:

$$
\nabla f = (2x, 2y, 2z)
$$

Okay, we know what a gradient is and also how to calculate it using Python. But how useful is it? What does it represent? How can we solve optimization problems with it?





Learn to generalize the derivative to solve problems with more than one variable.

Introduction

Derivatives and Gradients

First Optimization Algorithms

Population Methods

Adding Constraints

Solving Sudoku and the 8-Queens Puzzle as Constraint Satisfaction

Linear Constrained Optimization

Summary and Conclusion

Appendix

Maze Solver Using the Ant Colony Optimization Algorithm

Derivatives in Multiple Dimensions

Derivatives in multiple dimensions