Learn machine learning with scikit-learn, covering supervised learning, clustering, regression, SVMs, autoencoders, and ensemble methods through practical Python projects.

ml.tar.gz

Playground

Playground-SPA

Complexity CH-1

regulizer CH-1

regulizer CH-1-S

regulizer ch1_v2

complexity ch1_v2

code_widget

ST_kernel_trick-LIVE

python

pytorch

ST_Autoencoders

code_widget_torchsummary

code_widget_tqdm

SPA_tqdm

AE_multiple

SPA_server

Implementation of NN

c1l3

Implementation of NN-copy

This course focuses on core concepts, algorithms, and machine learning techniques. It explores the fundamentals, implements algorithms from scratch, and compares the results with scikit-learn, the Python machine learning library. This course contains examples, theoretical knowledge, and codes for various ML algorithms.

You’ll start by learning the essentials of machine learning and its applications. Then, you’ll learn about supervised learning, clustering, and constructing a bag of visual words project, followed by generalized linear regression, support vector machines, logistic regression, ensemble learning, and principal component analysis. You’ll also learn about autoencoders and variational autoencoders and end with three exciting projects.

By the end, you’ll have a solid understanding of machine learning and its algorithms, hands-on experience implementing such algorithms and applying them to different problems, and an understanding of how each algorithm works with the provided examples.

Fundamentals of Machine Learning: A Pythonic Introduction

# Hard-margin SVM
**Hard-margin SVM** is a type of support vector machine that aims to find the maximum margin hyperplane that perfectly separates the two classes without allowing any misclassification. It’s useful when the dataset is linearly separable and there are no outliers. 

# Linearly separable case
Given a binary classification dataset $D=\{(\bold x_1, y_1), (\bold x_2, y_2), \dots,(\bold x_n, y_n)\}$, where $\bold x_i \in \R^d$ and $y_i \in \{-1, 1\}$, if a hyperplane $\bold w^T\phi(\bold x)=0$ exists that separates the two classes, the dataset is said to be **linearly separable** in the feature space defined by the mapping $\phi$. We’ll assume for now that the dataset $D$ is linearly separable. The goal is to find the hyperplane with a maximum margin by optimizing the following objective:

$$
\max_{\bold w}\bigg[\frac{1}{\|\bold w\|}\min_{i}\big(y_i\bold w^T\phi(\bold x_i)\big)\bigg] 
$$

The direct optimization of the above objective can be very complex. Here's a derivation of an equivalent optimization problem which is easier to solve:
 
$$
\begin{align*}

\max_{\bold w}\bigg[\frac{1}{\|\bold w\|}\min_{i}\big(y_i\bold w^T\phi(\bold x_i)\big)\bigg] 

& = \max_{\bold w}\frac{1}{\|\bold w\|}\bigg[\min_{i}\big(y_i\bold w^T\phi(\bold x_i)\big)\bigg] \\

& = \max_{\bold w,\gamma}\frac{\gamma}{\|\bold w\|}

\;\;\;\;\;\;\;\;s.t. \;\;\;\;\;\;\; y_i\bold w^T\phi(\bold x_i)\ge\gamma \;\;\;\;\;\;\; \forall_i\\

& = \max_{\bold w,\gamma}\frac{1}{\|\bold w/\gamma\|} 

\;\;\;\;s.t. \;\;\;\;\;\;\; y_i(\bold{w}/\gamma)^T\phi(\bold x_i)\ge 1 \;\;\;\; \forall_i\\

& = \max_{\bold w^{'}}\frac{1}{\|\bold w^{'}\|} 

\;\;\;\;\;\;\;s.t. \;\;\;\;\;\;\; y_i(\bold{w}^{'T}\phi(\bold x_i)\ge 1 \;\;\;\;\;\;\; \forall_i\\

& = \min_{\bold w^{'}}\|\bold w^{'}\| 

\;\;\;\;\;\;\;\;\;s.t. \;\;\;\;\;\;\; y_i(\bold{w}^{'T}\phi(\bold x_i)\ge 1 \;\;\;\;\;\;\; \forall_i\\

& = \min_{\bold w^{'}}\|\bold w^{'}\|^2 

\;\;\;\;\;\;\;s.t. \;\;\;\;\;\;\; y_i(\bold{w}^{'T}\phi(\bold x_i)\ge 1 \;\;\;\;\;\;\; \forall_i\\

\end{align*}
$$

Here, $\bold w^{'}=\bold w/\gamma$. 

This scaling allows us to consider the support vectors to have a distance of $1$ and all other vectors to have a distance of more than $1$.

## Mathematical explanation

In the derivation, we introduced the variable $\gamma$ as $\gamma = \min_i(y_i\bold w^T\phi(\bold x_i))$, that is, $\gamma$ is the minimum value of $y_i\bold w^T\phi(\bold x_i)$. The reason for introducing $\gamma$ is to make the optimization problem easier to handle. Without $\gamma$, the objective function involves a minimum of overall data points, which makes it difficult to optimize. Introducing $\gamma$ allows us to remove the outer minimum and simplify the objective function.

By introducing $\gamma$, we can rewrite the objective function as:
$$
\begin{align*}
\max_{\bold w}\frac{\gamma}{\|\bold w\|}
\end{align*}
$$

Dividing both sides by $\gamma$ doesn't change the direction of the inequality since $\gamma$ is positive, and we can safely introduce a new variable $\bold w' = \bold w/\gamma$ to remove $\gamma$ from the objective. After that, we convert our problem to a minimization problem by taking the reciprocal of $\|\bold w\|$. When we convert a maximization problem into a minimization problem, it doesn't affect the constraints because the constraints only specify the conditions that need to be satisfied regardless of whether the objective function is being maximized or minimized. 
Finally, we take the square of the norm, which is more convenient for optimization.

Learn how to implement and optimize the hard-margin SVM.

Hard-Margin SVM

Course Overview

Supervised Learning

Detect Cyber Intrusion Using Machine Learning

Clustering

Project: Bag of Visual Words

Generalized Linear Regression

Face Recognition Using Kernel Linear Discriminant

Support Vector Machine

Logistic Regression

Ensemble Learning

Early Stage Diabetes Prediction Using Ensemble Learning

Decoding Dimensions: PCA and Autoencoders

Image Reconstruction Using PCA

Image Colorization using Autoencoders

Colorful Face Generation with VAEs

Appendix

Wrapping Up

Hard-Margin SVM

Hard-margin SVM

Linearly separable case