Ridge and Lasso Regression

Without regularization, we only have to optimize for training loss, but with regularization, we need to optimize for both training loss (data) and the regularization function (smoothness).

Ridge regression

When the model $f_\bold w$ is linear in parameters, the loss function $L$ is squared loss, and the regularization function $R$ is L2 norm, then the regression problem is known as ridge regression.

Linear model

Let $D=\{(\bold x_i,y_i) | 1 \le i \le n\}$ be a training data set for regression with a single real target, that is, $y_i \in \R$. When the number of features is $d$, that is, $\bold x_i=(x_{i1},x_{i2},\dots,x_{id})$ ...