Building Machine Learning Classification Models with Python

The idea of classification permeates every aspect of our daily life. We categorize things continuously depending on their traits and characteristics, whether we’re doing laundry by color or identifying species of birds. We frequently take classification for granted since it is ingrained in our mental processes. However, classifying things accurately and efficiently is essential for making informed decisions and navigating the complexities of real-world problems.

The classification process, a key concept in the journey to learn machine learning, assigns a label or category to a given input based on its traits or attributes. The task of predicting the class or category of a new observation based on its similarity to previously observed examples is known as classification in machine learning and statistics.

For instance, the classification task in email spam filtering is to decide whether an incoming email is spam. A machine learning algorithm may evaluate the email’s content, sender, and other features to make this conclusion. It will then categorize the email as spam or not based on patterns it discovered from previously labeled cases.

Loading the dataset#

Let’s start by building a classification model. We’ll take an example of an Iris dataset that contains measurements of different attributes of three species of iris flowers. Begin by clicking the “Run” button to display the top few rows of the DataFrame.

• Line 2: We import the pandas library to read the DataFrame.
• Lines 5–8: We import the load_iris function from the sklearn.dataset module. This allows us to load the Iris dataset on line 8.
• Lines 11–13: We create a DataFrame using the features and target data from the Iris dataset.
• Line 15: We print five randomly selected rows of the DataFrame using the sample() function.

The DataFrame contains $N=3$ iris species classes, “setosa,” “virginica,” and “versicolor,” with the following four features:

• Sepal length: The length of the sepal in centimeters.
• Sepal width: The width of the sepal in centimeters.
• Petal length: The length of the petal in centimeters.
• Petal width: The width of the petal in centimeters.

In machine learning, features are the independent variables, and target is the dependent variable. Our goal is to build a classification model in Python that predicts the flower’s species using the four features as input.

Classification models with Python#

In this blog, we will focus on logistic regression. Logistic regression is a method that statistically models a binary classification task. It predicts the probability $p$ that the input features fall into a specific class.

Mathematically, we model the logistic regression model as follows:

$$p = 1 / (1 + e^{-z}).$$

Here, $z$ defines the weighted linear combination of the input features and is calculated as follows:

$$z = w_0 + w_1 x_1 + w_2 x_2 + ... + w_n x_n.$$

The linear regression algorithm, such as gradient descent, finds the optimal values for the weights that maximize the likelihood of the observed data.

Let’s see how this can be done using Python:

• Lines 4–5: We import LogisticRegression and train_test_split from the sklearn library.

• Line 14: We split the features X and target y into training and test datasets. The training dataset trains the model, while the test dataset evaluates its performance.

• Lines 17–20: We create a logistic regression model and train the classifier on training data X_train and y_train.

Validation#

Now that we have created and trained a classification model using the training data, we will proceed toward evaluating the model.

We will start by defining the confusion matrix. A confusion matrix is a $N\times N$ matrix used to evaluate a classification model’s performance and compares the model’s prediction and actual labels. The elements of a confusion matrix are defined as follows:

• True positive (TP): The number of instances that belong to species of class $i$ and are correctly predicted by the model as class $i$.

• False positive (FP): The number of instances that do not belong to species of class $i$ but are incorrectly predicted by the model as class $i$.

• False negative (FN): The number of instances that belong to species of class $i$ but are incorrectly predicted by the model as a different class.

• True negative (TN): The number of instances that do not belong to species of class $i$ and are correctly predicted by the model as not belonging to class $i$.

Let’s see how to calculate the confusion matrix in Python for our Iris dataset:

Here, the model.predict() function is applied to the test data on line 6, and a confusion matrix is calculated on line 9 using the confusion_matrix() function. The diagonal elements in the confusion matrix show the values of TP.

Now it will be easier to define the following performance metrics.

• Accuracy: A ratio of the sum of TP and TN to the total predictions. Accuracy tells us about the overall correctness of the model’s prediction. A higher accuracy shows a better-performing model.

• Precision: A ratio of TP to the sum of TP and FP. Precision is important when the cost of a false positive event is high. A higher precision tells us that the species predicted as class $i$ by our model are more likely to be correctly identified as class $i$.

• Recall: A ratio of TP to the sum of TP and FN. The recall is important when the cost of FN is high and focuses on the model’s ability to avoid FN.

Now that we understand the performance metrics let’s calculate our model’s accuracy, precision, and recall on the test dataset.

Here, we evaluate the model’s performance using the expected output in y_test, and the model predicted output in y_pred.

• On line 6, the accuracy of the model is calculated using metrics.accuracy_score function.
• On line 7, the precision of the model is calculated using metrics.precision_score function.
• On line 8, the recall of the model is calculated using metrics.recall_score function.

The average parameter in calculating precision and recall on lines 7–8 is set to macro. This individually computes the corresponding values of each class and then averages them.

The results show that our classification model fits well and provides accurate predictions.

This blog has briefly introduced a logistic regression classification model with Python. We encourage you to explore other classification models like the random forest classifier, support vector machine (SVM), K-nearest neighbors (KNN), and decision trees to build accurate and robust models. Additionally, you can check out the following courses on Educative: