**Regularization** is a form of regression. In regularization, the estimated coefficients in regression equations are constrained to zero. This is one of the major techniques used to avoid overfitting. 

Regularization is a method that can be utilized for any given training set. It reduces errors by fitting the function appropriate to the training set. As a result, this method avoids overfitting. 

# Regularization techniques

The commonly used regularization techniques are:




1. **Lasso Regression (L1 regularization)** calculates the absolute value of the coefficients' magnitude and adds it to the loss function.

General equation looks like:
```xml
y = β0 + β1x1 + β2x2 + ··· βkxk + λ(slope)²
```
Loss function:
$||w||_1 = |w_1| + |w_2| + ... |w_N|$

2. **Ridge Regression (L2 regularization)** calculates the square value of the coefficients' magnitude and adds it to the loss function.
The general equation looks like:
```xml
y = β0 + β1x1 + β2x2 + ··· βkxk + λ|slope|
```

Loss function:
$||w||_2 = (|w_1|^2 + |w_2|^2 + ... |w_N|^2)^{1/2}$


Output function:

$y^{hat} = w_1x_1 + w_2x_2 + ... w_Nx_N + b$


# Loss function in regularized regression
We define the loss function in logistic regression as:
```xml
L(y_hat,y) = y log y_hat + (1 - y)log(1 - y_hat)
```
 
Loss function with no regularization:
```xml
L = y log (wx + b) + (1 - y)log(1 - (wx + b))
```   
Let's say the data overfits the above function.

Loss function with L1 regularization:
```xml
L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w||
```    
Loss function with L2 regularization:
```xml
L = y log (wx + b) + (1 - y)log(1 - (wx + b)) + lambda*||w|| ^ 2   
```
Note that lambda is a hyperparameter known as the regularization constant -- it is greater than zero.

What is regularized regression?

Regularization techniques

Loss function in regularized regression