Learn to solve optimization problems in Python using essential math tools, metaheuristic methods, and constrained optimization techniques. Master efficient solutions for real-world applications.

Optimization with Python from zero to hero.png

cvxpy.tar.gz

cvxpy

Optimization theory seeks the best solution, which is pivotal for machine learning, cost-cutting in manufacturing, refining logistics, and boosting finance profits.

This course provides a detailed description of different optimization problems and the techniques used to solve them. You’ll begin with the formal definition of an optimization problem and an overview of essential mathematical tools: derivatives, gradients, and Hessian. With this knowledge, you’ll implement solutions for several optimization problems. You’ll also learn approximate metaheuristic population methods like particle swarm optimization and genetic algorithms, and solve constrained optimization problems using techniques like penalty methods and constraint simplification. Finally, you’ll solve linear programs and integer programs. 

By the end of this course, you’ll become proficient in formulating different kinds of problems as optimization problems and become skilled in the Python tools that efficiently solve these problems.

Mastering Optimization with Python

**Linear optimization** is the branch of optimization that deals with problems in which the target function and the constraints are linear. The field that studies the algorithms and techniques that exploit the linearity of optimization problems is called **linear programming**.

# Linear optimization

What do we mean by linear? A linear function $f$ has the following form:

$$
f(x_1, x_2, ..., x_n) = a_1x_1 + a_2x_2 + ... + a_nx_n + b
$$

A **linear function** is a polynomial function of degree zero or one. That means that the exponent of each variable is either zero or one. For cases in which the function only has one variable, its graph is a straight line. In the general case of $n$ variables (as in the example above), the graph of the function is what is known in math as a hyperplane.

In the example above, $a_1, a_2, ..., a_n$ and $b$ are **scalars** (i.e., numbers). That means that linear functions only allow multiplications by scalars and sums. So a linear optimization problem looks like this:

$$
\min_{x_1, x_2, ..., x_n} a_1x_1 + a_2x_2 + ... + a_nx_n + b \\
s.t.: p_{i1}x_1 + p_{i2}x_2 + ... + p_{in}x_n < p_i \\
i = 1, 2, ..., m 
$$

In this example, there are $m$ constraints. All the constraints should be linear as well. All $p_{ij}, p_i, q_{ij}, q_i$ are scalars. We use $p$ and $q$ to differentiate between inequality and equality constraints. The first two constraints are inequalities, and the last two are equalities.

>**Note:** The left part of the constraints just has scalar multiplications and sums while the right part is just a scalar. That's why we say the constraints are linear.

# Matrix form

This way of representing a linear problem allows us to understand it better if we don't have much experience with matrix notation. However, we'll use vectors and matrices from now on. In general, a linear optimization problem can be defined as follows:

$$
\min_{\mathbf{x}} \mathbf{a}\mathbf{x}^\top \\
s.t.: P\mathbf{x}^\top < \mathbf{p}\\
Q\mathbf{x}^\top = \mathbf{q}
$$

Both definitions are equivalent. Here, $\mathbf{x} = (x_1, x_2, ..., x_n)$, $\mathbf{a} = (a_1, a_2, ..., a_n)$, $\mathbf{p} = (p_1, p_2, ..., p_m)$, and $\mathbf{q} = (q_1, q_2, ..., q_k)$. The number of inequality constraints doesn't have to be equal to the number of variables or the number of equality constraints. Matrix $P$ has dimensions $m \times n$ and $P_{ij} = p_{ij}$, while matrix $Q$ has dimensions $k \times n$ and $Q_{ij} = q_{ij}$.

Now we can translate any linear problem described as sums and scalar multiplications to a matrix notation definition. The tools we're going to use to solve these problems use matrix notation.

Learn what linear optimization problems are.

Introduction

Derivatives and Gradients

First Optimization Algorithms

Population Methods

Adding Constraints

Solving Sudoku and the 8-Queens Puzzle as Constraint Satisfaction

Linear Constrained Optimization

Summary and Conclusion

Appendix

Maze Solver Using the Ant Colony Optimization Algorithm

Introduction to Linear Optimization

Linear optimization