Introduction to Regression Methods

Regression refers to the task of predicting a continuous class variable, given some input features. We’ll discuss linear regression as the main method frequently used for regression, and some other more advanced techniques in this chapter.

For the sake of consistency, we are going to use the following system of naming:

• $X=(X_1,X_2,X_3,...,X_n)$ are the input variables.
• $x = (x_1, x_2, x_3, . . . , x_n)$ is an input instance, in which input variable $X_i$ takes on value $x_i$. To denote different input instances, we use $x^{(i)}$, $x^{(j)}$.
• Here, $Y$ is the target/output/class variable and $y$ is a value $Y$ can take.
• In case there are many class values to be discussed, we’ll use subscripts to differentiate them (e.g., $y_j$ or $y_k$). Similar to the input, we use $y^{(i)}$, $y^{(j)}$ to denote corresponding target values of inputs $x^{(i)}$, $x^{(j)}$.

Linear regression

Linear regression is a regression method that models the relationship between a continuous class/target variable $Y$, called the dependent variable, and a set of input features $X = (X_1, X_2, . . . , X_n)$, called independent variables. A dependent variable is what we are measuring (or predicting in this case) and an independent variable is what we are controlling or manipulating (or have as an input in this case). Note that we also call the input features, $X$, feature vector, since it comprises multiple variables, each corresponding to one dimension in the vector space formed by the features. As such, we are going to use data points and data vectors interchangeably in this chapter.

Back to the method at hand, linear regression assumes a linear relationship between the variables. The dependent variable $Y$ can be expressed as a linear combination of independent variables $X_1, X_2,\cdots, X_n$, as illustrated in the below figure.