Introduction to Linear Regression
Learn about linear regression, the no free lunch theorem, and bias-variance trade-offs.
This lesson will discuss linear regression, which is a straightforward approach and is considered the "workhorse" approach for supervised machine learning. Linear regression has been around for a long time and is the topic of countless textbooks. It's a practical and widely used statistical or machine learning model. Moreover, it serves as a good jumping-off point for newer approaches.
History
The earliest form of regression was the least squares method, which Legendre published in 1805 and was later published by Gauss in 1809. However, the term "regression" was coined by Sir Francis Galton in his work, published in 1875, while he was describing the biological phenomenon of relating the heights of descendants to their tall ancestors. For Sir Galton, regression had only this biological meaning, but Udny Yule and Karl Pearson later extended his work to a more general statistical context.
In his study, Sir Galton discovered that a man's son tends to be roughly as tall as his father, but the son's height tends to be closer (regress or drift toward) to the overall average height.
Let's consider the simplest possible example:
We have two data points, and , as shown in the diagram. If we draw a line as close as possible to each point, the line will pass through the points.
To fit a line using the least squares method (classical linear regression and a standard approach), we only measure the closeness in the up and down directions.
Let’s add two more data points, and :
Now, if we draw a line following the same rule, the least squares method, the line will be different for the given points. It may not pass through the data points.
The questions are:
- Is the black line the best-fitted line for the given data points?
- If yes, why?
Let’s explore a little more to learn about the best-fitted line.
Following the standard approach, the best fit in the least squares approach minimizes the sum of squared