The Motivation for Regularization

What is regularization?

Regularization is a technique used in machine learning to prevent overfitting and improve the generalization of the trained model. The main idea behind regularization is to add a penalty term to the cost function of the model that discourages the model from learning complex or redundant features that are specific to the training data and might not generalize well to new, unseen data.

The bias-variance trade-Off

We can extend the basic logistic regression model that we have learned about by using regularization, also called shrinkage. In fact, every logistic regression that you have fit so far in scikit-learn has used some amount of regularization. That is because it is a default option in the logistic regression model object. However, until now, we have ignored it.

As you learn about these concepts in greater depth, you will also become familiar with a few foundational concepts in machine learning: overfitting, underfitting, and the bias-variance trade-off. A model is said to overfit the training data if the performance of the model on the training data (for example, the ROC AUC) is substantially better than ...