Nonlinear Regression

Going beyond linear models

Linear regression is often applied because many relations in practical applications can be approximated by a linear model. It is also simple and offers a good starting point for experimenting with new algorithms. Linear models are, therefore, still important and should be considered a good first step in modeling data. However, going beyond linear models is one of the gifts of modern machine learning. Allowing nonlinear relations in a model opens the modeling space to an infinite number of possible models and hence the possibility of an unbounded number of parameters. Overfitting is thus often an even more pronounced problem in high-dimensional nonlinear models. Solving, or at least mediating this problem, is therefore strongly tied to the success of the models, such as deep learning.

Transforming a nonlinear problem into a linear problem

As a starting point for discussing such cases, let’s first consider a polynomial of order $n$.