R-square and Goodness of the Fit

Let’s understand the r-square and the goodness of fit.

R-squared

The coefficient of determination, $R^2$ or $r^2$ (“R-squared”), is the proportion of the variance in the dependent variable that is predictable from the independent variable(s). It measures how well-observed outcomes are replicated by the model based on the proportion of total variation of outcomes explained by the model.

The better the linear regression (on the right) fits the data in comparison to the simple average (on the left graph), the closer the value of $R^2$ is to 1. The areas of the blue squares represent the squared residuals concerning the linear regression. The areas of the red squares represent the squared residuals concerning the average value.

Key limitations of R-squared:

• Can’t determine whether the coefficient estimates and predictions are biased, so we must assess (evaluate) the residual plots.

• Does not indicate whether a regression model is adequate. We can have a low R-squared value for a good model or a high R-squared value for a model that does not fit the data.

• It’s possible to get a negative value of R-squared. If the fit is worse than a horizontal line, the R-squared is negative. In this case, the R-squared can’t be interpreted as the square of a correlation. Such situations indicate that a constant term should be added to the model.

Typically, the accuracy score $R^2$,

$$\small \begin{align*} R^2 &= 1- \frac{\text{Explained variance}}{\text{Total variance}}\\ &= 1 - {\text{fraction of unexplained variance}}\\ &= 1 - \frac{\sum_{i=1}^n(y_i - \hat{y}_i)^2}{\sum_{i=1}^n(y_i - \bar{y})^2} \end{align*}$$ ...