Nonparametric Regression

We worked on parametric approaches in the previous lesson. This lesson is dedicated to the nonparametric approach for regression.

Using the appropriate value of parameters, we can fit the data. But what if fitting the data is not possible with one model?

See the example below.

We can think of fitting a simple regression line on this data. It will look similar to this.

However, this is not ideal. Maybe we can try high order polynomials. See the example of fit below.

It is better than the previous solution, but it still does not capture the best model. What if we can fit the data locally (small points) instead of trying globally?

This is a good and reasonable fit. The only limitation is that we need a good amount of data to fit these local models.

Nearest-neighbor regression

In this technique, we predict the value based on past nearest data. We find the most similar K values to the new value and take their observations to predict the value by a simple average or weighted average. Consider the below example.

Assume that all the blue points are training data and we have to predict a green point. We can use the nearest-neighbor regression to predict the value. Consider K=1. So, we predict the value of the nearest target value of its neighbor.

This is called 1-nearest neighbor regression.

Instead of 1-nearest neighbor, K neighbors are considered. This will give a more generalized prediction and safety from outliers. We can also provide weights on neighbors to get weighted nearest neighbors.

Distance metrics

We choose the neighbor based on the minimum distance from the new point. We use different types of metrics to get this distance. A few examples are correlation-based, rank-based, cosine similarity, Euclidean, etc.

Kernel regression

While we can weigh neighbors in the K-nearest neighbor regression. In kernel regression, we weigh all the data points in the dataset.

$$\widehat{y}=\frac{\sum _{i=1}^{i=N}Cei{y}_i}{\sum _{i=1}^{i=N}Cei}=\frac{\sum _{i=1}^{i=N}Kernel(distance(x_i{x}_e)\ast y_i}{\sum _i^{}}$$ ...