Introduction to Regularization

Let's start with understanding the difference between underfitting and overfitting.

Underfitting vs. overfitting

Let's say we have the following plots (shown below):

• Left plot: Underfitted model with low accuracy score (R-squared) and higher error (SSE: sum of squared error).

• Middle plot: Model with average accuracy score and error. Can this be improved?

• Right plot: The overfitted model has a very high accuracy score and low error (practically R-squared = 1 and SSE = 0 since the fitted line passes through all the data points).

The middle and right plots are fitted with polynomial regression for a single predictor/feature (x). These models are nonlinear in the feature (x) space but linear in the parameters space. Although these models allow for a nonlinear relationship between the target (y), and the feature (x), polynomial regression is still considered linear because it is linear in the regression coefficients.

Coefficients $\beta$ are also represented by weights $w$ in literature. A regression based on standardized variables typically produces standardized (regression) coefficients ...