Regularized Regression

Overfitting is when the model gives low error on training data but a high error on testing data. You learned about overfitting in the previous lesson. A highly complex model leads to overfitting. In regression, overfitting refers to large values of coefficients. When the value of coefficients is very high, that coefficient dominates, leading to overfitting.

So, we know overfitting is caused by large coefficients in regression. In regression, our objective is to reduce the cost. If, somehow, we can penalize the high coefficient value, we can get a better model. So, now we have to form a loss function that includes:

• How well our function is fitting the data
• The magnitude of the coefficients generated

Our cost would be the sum of the above two terms

$Total\; cost = Residual \;Sum \;of\; Square + Measure \;of\; the\; magnitude\; of \;coefficients$ ...