What is accuracy in machine learning?

Accuracy is the most widely known performance measurement metric for machine learning models. It is a relatively easy measuring metric, but it is often difficult to understand its credibility based on different scenarios. It is used in classification problems to identify the correct predictions a model makes regarding percentages.

Accuracy score

It is the value obtained when we divide the total number of correct predictions the model made by the total number of prediction models made, including the incorrect ones. Acccuracy helps to identify if the applied model is suitable for the available dataset.

Types of classifications

Different real-life cases are evaluated through the accuracy metric. Different classifications have different methods to measure the accuracy. Let's discuss the three basic types of classification: binary classification, multiclass classification, and multilabel classification.

Accuracy in binary classification

In binary classification, the instances are categorized under two classes only based on their attributes and properties, i.e., positive and negative. The goal is to separate the positive and negative cases and then check the accuracy to evaluate how many instances are correctly categorized. We use a simple formula for this calculation that divides the correctly classified cases from the total classified cases.

$Accuracy = (TP + TN) / (TP + TN + FP + FN)$

Where:

• TP: positive classes that are correctly predicted as positive.

• FN: positive classes that are incorrectly predicted as negative.

• TN: negative classes that are correctly predicted as negative.

• FP: negative classes that are incorrectly predicted as positive.

Real-life application

Different scenarios require the classification of instances under either a positive class or a negative class to make it clear if the instances have a particular property or not.

Accuracy in multi-class classification

In multi-class classification, the instances are categorized under multiple mutually exclusive classes based on their attributes and properties, where each instance can be associated with one class only. For example, if there are three primary colors balls, we can classify them under the red, blue, and yellow class, whereas the yellow ball instance can only be classified under yellow. We use the following formula for this calculation that returns true for the instances whose prediction output matches the true output label.

$Multiclass Accuracy = 1/n \sum _{1}^{n} [[t_i == p_i]]$

Where:

• $n$: total number of samples used for calculating accuracy.

• $t_i$ true output label of the given sample.

• $p_i$: predicted output label of the given sample.

• $[[...]]$: inversion brackets which return 1 for true expression otherwise 0.

Real-life application

Different scenarios require the classification of instances under multiple classes to make it easier to identify the instance based on its properties.

Note: Learn more about implementation through an image classification example.

Accuracy in multi-label classification

In multi-label classification, the instances are categorized under multiple mutually non-exclusive classes based on their properties, where each instance can be associated with multiple classes. It has a more balanced metric because there are more relaxed class categorizations available in it. We use the following formula for this calculation that returns and divides the instances whose prediction output matches the true output label out by all the instances for which an output label exists.

$Multilabel Accuracy = 1/n ∑_{1}^{n} (|t_i ∩ p_i| / |t_i ∪ p_i|)$

Where:

• n: total number of samples used for calculating accuracy.

• t_i ∩ p_i: samples for which the true and predicted output label are the same.

• t_i ∪ p_i: All the samples with a true or a predicted output label.

Real-life application

Different scenarios require classifying instances under multiple labels to make it easier to find instances based on their attributes.

Limitation of accuracy

Accuracy works best on balanced datasets and possibly gives a faulty result on an imbalanced dataset. For example, if the ratio of a fault occurrence and no-fault occurrence is 1: 99, then the accuracy will be 99%, ignoring that 1%. In this case, accuracy is a false indicator of model health.

Test your understanding

Match the classification types to the real-life scenarios they are used in.