Learn optimization for machine learning, including gradients, convex optimization, and gradient descent. Explore constrained optimization and advanced methods using NumPy and SciPy.

Optimization for Machine Learning with NumPy and SciPy copy.png

docker.tar.gz

In this course, you will learn about optimization, one of the fundamental pillars of mathematics and machine learning. Machine learning depends heavily on optimization because it allows the model to learn from data and generate precise predictions.

You will begin by introducing optimization. Then, you will learn about optimization basics, including gradients and integrals. Next, you will cover convex optimization. You will then learn how to compute gradient descent for non-convex optimization. Next, you’ll learn how to perform constrained optimization. You will finish the course by studying the miscellaneous methods of optimization, like Newton’s methods, quasi-Newton methods, and conjugate gradient descent.

After completing this course, you’ll have the practical skills to formulate, analyze, and implement optimization algorithms for machine learning using the NumPy and SciPy libraries. This will help you become a highly proficient data scientist or machine learning engineer.

Optimization for Machine Learning with NumPy and SciPy

$x_2$ is the entering variable and $x_1$ is the leaving variable.

$x_1$ is the entering variable and $x_2$ is the leaving variable.

$x_3$ is the entering variable and $x_1$ is the leaving variable.

$x_3$ is the entering variable and $s_1$ is the leaving variable.

In the simplex tableau below, which variables will be the entering and leaving variables?

|         | $x_1$     | $x_2$ | $x_3$ | $s_1$ | **RHS** |
|--------------|-----------|------------|------------|------------|------------|
| $x_1$ | 1      | 3        | -2        | 1        | 36
| $s_1$      | 0 | 1     | 1     | 0      | 24
| $z$     | 0  | -4     | -3      | 1      | 198

In the simplex tableau below, which variables will be the entering and leaving variables?
<table>
<thead>
<tr>
<th></th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math>x1​</th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">x_2</annotation></semantics></math>x2​</th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>3</mn></msub></mrow><annotation encoding="application/x-tex">x_3</annotation></semantics></math>x3​</th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">s_1</annotation></semantics></math>s1​</th>
<th>RHS</th>
</tr>
</thead>
<tbody>
<tr>
<td><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math>x1​</td>
<td>1</td>
<td>3</td>
<td>-2</td>
<td>1</td>
<td>36</td>
</tr>
<tr>
<td><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">s_1</annotation></semantics></math>s1​</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>24</td>
</tr>
<tr>
<td><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi></mrow><annotation encoding="application/x-tex">z</annotation></semantics></math>z</td>
<td>0</td>
<td>-4</td>
<td>-3</td>
<td>1</td>
<td>198</td>
</tr>
</tbody>
</table>

$\mathcal{D}(\lambda) = -\frac{\lambda^2 a}{4} - b\lambda$

$\mathcal{D}(\lambda) = \frac{\lambda^2 a^2}{4}+  b\lambda$

$\mathcal{D}(\lambda) = -\frac{\lambda^2 a^2}{4} - b\lambda$

$\mathcal{D}(\lambda) = -\frac{\lambda a^2}{4} - b\lambda^2$

What is the dual objective of the following primal problem?

$$\min_x \  x^2 \ \text{ subject to } \ ax \leq b$$

What is the dual objective of the following primal problem?
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><munder><mrow><mi>min</mi><mo>⁡</mo></mrow><mi>x</mi></munder><mtext> </mtext><msup><mi>x</mi><mn>2</mn></msup><mtext>  subject to  </mtext><mi>a</mi><mi>x</mi><mo>≤</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">\min_x \ x^2 \ \text{ subject to } \ ax \leq b
</annotation></semantics></math>xmin​ x2  subject to  ax≤b

In the simplex tableau below, which variable will be the entering variable?

|         | $x_1$     | $x_2$ | $x_3$ | $s_1$ | **RHS** |
|--------------|-----------|------------|------------|------------|------------|
| $x_1$ | 1      | 3        | -2        | 1        | 36
| $s_1$      | 0 | 1     | 1     | 0      | 24
| $z$     | 0  | 4     | 3      | 1      | 198

In the simplex tableau below, which variable will be the entering variable?
<table>
<thead>
<tr>
<th></th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math>x1​</th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>2</mn></msub></mrow><annotation encoding="application/x-tex">x_2</annotation></semantics></math>x2​</th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>3</mn></msub></mrow><annotation encoding="application/x-tex">x_3</annotation></semantics></math>x3​</th>
<th><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">s_1</annotation></semantics></math>s1​</th>
<th>RHS</th>
</tr>
</thead>
<tbody>
<tr>
<td><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>x</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">x_1</annotation></semantics></math>x1​</td>
<td>1</td>
<td>3</td>
<td>-2</td>
<td>1</td>
<td>36</td>
</tr>
<tr>
<td><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><msub><mi>s</mi><mn>1</mn></msub></mrow><annotation encoding="application/x-tex">s_1</annotation></semantics></math>s1​</td>
<td>0</td>
<td>1</td>
<td>1</td>
<td>0</td>
<td>24</td>
</tr>
<tr>
<td><math xmlns="http://www.w3.org/1998/Math/MathML"><semantics><mrow><mi>z</mi></mrow><annotation encoding="application/x-tex">z</annotation></semantics></math>z</td>
<td>0</td>
<td>4</td>
<td>3</td>
<td>1</td>
<td>198</td>
</tr>
</tbody>
</table>

$$\pounds = x_1 \log(x_1) + x_2 \log(x_2)  + \lambda_1 (x_1 + x_2 - 1) - \lambda_2 (x_1 + x_2 - 1) + \lambda_3 (x_1 - 0.1) + \lambda_4 (x_2 - 0.2)$$

$$\pounds = x_1 \log(x_1) + x_2 \log(x_2)  + \lambda_1 (x_1 + x_2 - 1) - \lambda_2 (x_1 - 0.1) - \lambda_3 (x_2 - 0.2)$$

$$\pounds = x_1 \log(x_1) + x_2 \log(x_2)  + \lambda_1 (x_1 + x_2 - 1) - \lambda_2 (x_1 + x_2 - 1) - \lambda_3 (x_1 - 0.1) - \lambda_4 (x_2 - 0.2)$$

$$\pounds = x_1 \log(x_1) + x_2 \log(x_2)  + \lambda_1 (x_1 + x_2 - 1) + \lambda_3 (x_1 - 0.1) + \lambda_4 (x_2 - 0.2)$$

Which is the correct Lagrangian function for the following primal problem?
$$\min_x \ x_1 \log(x_1) + x_2 \log(x_2) \\ \text{ subject to } \\ x_1 + x_2 = 1 \\ x_1 \geq 0.1, x_2 \geq 0.2

Which is the correct Lagrangian function for the following primal problem?
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><munder><mrow><mi>min</mi><mo>⁡</mo></mrow><mi>x</mi></munder><mtext> </mtext><msub><mi>x</mi><mn>1</mn></msub><mi>log</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>x</mi><mn>1</mn></msub><mo stretchy="false">)</mo><mo>+</mo><msub><mi>x</mi><mn>2</mn></msub><mi>log</mi><mo>⁡</mo><mo stretchy="false">(</mo><msub><mi>x</mi><mn>2</mn></msub><mo stretchy="false">)</mo><mspace linebreak="newline"></mspace><mtext> subject to </mtext><mspace linebreak="newline"></mspace><msub><mi>x</mi><mn>1</mn></msub><mo>+</mo><msub><mi>x</mi><mn>2</mn></msub><mo>=</mo><mn>1</mn><mspace linebreak="newline"></mspace><msub><mi>x</mi><mn>1</mn></msub><mo>≥</mo><mn>0.1</mn><mo separator="true">,</mo><msub><mi>x</mi><mn>2</mn></msub><mo>≥</mo><mn>0.2</mn></mrow><annotation encoding="application/x-tex">\min_x \ x_1 \log(x_1) + x_2 \log(x_2) \\ \text{ subject to } \\ x_1 + x_2 = 1 \\ x_1 \geq 0.1, x_2 \geq 0.2
</annotation></semantics></math>xmin​ x1​log(x1​)+x2​log(x2​) subject to x1​+x2​=1x1​≥0.1,x2​≥0.2

**(Select all that apply.)** Which algorithm(s) can be used to solve the constraint optimization below? 

$$\min_x \  a_1^{x_1}\cdot a_2^{x_2} \cdot \cdot \cdot  a_n^{x_n} \ \text{ subject to } Ax \leq b$$


(Select all that apply.) Which algorithm(s) can be used to solve the constraint optimization below?
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block"><semantics><mrow><munder><mrow><mi>min</mi><mo>⁡</mo></mrow><mi>x</mi></munder><mtext> </mtext><msubsup><mi>a</mi><mn>1</mn><msub><mi>x</mi><mn>1</mn></msub></msubsup><mo>⋅</mo><msubsup><mi>a</mi><mn>2</mn><msub><mi>x</mi><mn>2</mn></msub></msubsup><mo>⋅</mo><mo>⋅</mo><mo>⋅</mo><msubsup><mi>a</mi><mi>n</mi><msub><mi>x</mi><mi>n</mi></msub></msubsup><mtext>  subject to </mtext><mi>A</mi><mi>x</mi><mo>≤</mo><mi>b</mi></mrow><annotation encoding="application/x-tex">\min_x \ a_1^{x_1}\cdot a_2^{x_2} \cdot \cdot \cdot a_n^{x_n} \ \text{ subject to } Ax \leq b
</annotation></semantics></math>xmin​ a1x1​​⋅a2x2​​⋅⋅⋅anxn​​  subject to Ax≤b

Evaluate your constrained optimization concepts via an MCQ quiz.

Introduction to Optimization

Vector Calculus

Convex Optimization

Gradient Descent for Non-Convex Optimization

Use Particle Swarm Optimizer to Optimize a Non-convex Function

Constrained Optimization

Miscellaneous Methods

Course Conclusion

Test Your Concepts of Optimization

Training Support Vector Machines (SVMs)

Quiz Yourself on Constrained Optimization