Simple Linear Regression

During the last stage of the data science lifecycle, we are faced with the question of which model to choose to make predictions. We can decide what kind of model to use by looking at the relationship between the data variables that we have.

Let’s take the example of predicting tips paid to waiters.

From the plot, we can see that there is a direct linear relationship between total_bill and tips where increasing/decreasing total_bill results in an increase/decrease in tips as well. Looking at the linear relationship, we need a linear model for this problem. If we represent total_bill values by $x$ then our prediction ($\hat{y}$) becomes

$$\hat{y} = \theta_0x + \theta_1$$

This is the form of a linear equation. The term $\theta_1x$ implies that increasing/decreasing total bill(x) will increase/decrease the tip($\hat{y}$). Now that we have our model, we need to fit it to the data using gradient descent optimization. We will be using the mean squared error loss function.

Optimizing the Simple linear model

If we denote our predictions by $\hat{y}$ and the actual values by $y$, then loss function will be calculated as:

$$L(\theta,X,Y) = \frac{1}{n}\sum_{i=1}^{n}{( y_i - \hat{y_i})^2}$$

$$L(\theta,X,Y) = \frac{1}{n}\sum_{i=1}^{n}{( y_i - (\theta_0x + \theta_1 ))^2}$$

We will be minimizing this loss function with gradient descent. Recall that in gradient descent we:

• Start with random initial value of $\theta$.
• Compute $\theta_t - \alpha \frac{\partial}{\partial \theta} L(\theta,X,Y)$