Naïve Bayes explained

Overview#

You might be familiar with the growing excitement around machine learning and its various applications. Amidst this frenzy, one algorithm stands out for its simplicity and effectiveness in classification tasks: the Naïve Bayes classifier. Its versatility makes it applicable in numerous real-world scenarios, including the following:

• Spam email detection: Based on the presence of specific words or phrases, Naïve Bayes is used to separate spam emails from valid ones.

• Sentiment analysis: This is a technique used in natural language processing applications, such as customer reviews or social media posts, to classify text into positive, negative, or neutral attitudes.

• Medical diagnosis: Based on test findings and patient symptoms, Naïve Bayes is used to forecast the likelihood that a specific disease is present.

• Recommendation systems: These are used in recommendation engines to forecast user preferences and make pertinent product or service recommendations in response to user activity.

• Document classification: News articles, academic papers, and legal documents are among the specified categories that can be classified using Naïve Bayes.

As seen from the examples above, the Naïve Bayes algorithm can work well with textual data as well as structured (or tabular) data. In this blog, our focus will be on structured data. We will see how Naïve Bayes works, along with exploring some of its advantages and disadvantages.

Bayes’ theorem #

Before delving further, let’s first take a look at the Bayes’ theorem, upon which the Naïve Bayes algorithm is based.

Bayes’ theorem is a key idea in probability theory and statistics that explains how to update the probability of a hypothesis (or event) in light of fresh information or data.

Bayes’ theorem mathematically expresses this relationship between event $A$ and event $B$ as follows:

Here:

• $P(A∣B)$, known as posterior probability, denotes the conditional probability of event $A$ occurring, given that event $B$ has previously occurred.

Note: Conditional probability is defined as the probability of an event ($A$) occurring, given that another event ($B$) has already occurred.

• $P(B∣A)$, also called likelihood, is the conditional probability of event $B$ happening, given that event $A$ has already occurred.

• $P(A)$, also known as prior probability, is the probability of event $A$ occurring based on prior history.

• Evidence probability, or $P(B)$, is the probability of the event happening.

Let’s explain Bayes’ theorem using a coin toss example.

Using Bayes’ Theorem, we derive the following equation:

We have all the values that can be plugged into the above equation. Therefore,

Naïve Bayes#

Now that we have a brief understanding of Bayes’ theorem, let’s take a look at the Naïve Bayes algorithm—a probabilistic algorithm that is based on the assumption that features are independent of one another. The assumption of independence among features means that the occurrence or value of one feature does not affect or depend on the occurrence or value of another feature. However, this assumption might not hold true in real-world applications.

How Naïve Bayes works#

Assume we have $k$ attributes, where $A_1$​ through $A_k$​ are attributes with distinct features. The class is $C$ and can have multiple distinct values. Let’s further suppose that a test example $d$ with observed attribute values $a_1$​ through $a_k$​ is provided, and $a_i$​ is the value of attribute $A_i$, where $i=1,\cdots k_i$. In essence, classification involves calculating the subsequent posterior probability. The prediction is the class $c_j$​ such that

is maximum.

By applying the Bayes’ theorem, we get:

The class prior probability is $P(C=c_j)$, which is easily determined. The likelihood that a class will exist without taking any attributes into account is known as its prior probability. It is computed as the percentage of occurrences that are associated with that particular class in the existing dataset.

The denominator ${P(A_1 = a_1, A_2 = a_2, …, A_k = a_k)}$ can be ignored because it will be the same for every probability.

Therefore, we only need to calculate:

This can be written as:

Now, how do we calculate $P(A_i = a_i| C = c_j)$? Simple! We calculate it as follows:

This can also be easily calculated by looking at the dataset.

Finally, we have all the details. Therefore, to determine the most likely class for the test instance given a test example $d$, we compute the following:

The Naïve Bayes algorithm#

Let’s now take a look at the algorithm step by step followed by a working example.

1. Calculate prior probabilities: Calculate the prior probabilities of each class in the given dataset.

2. Calculate likelihoods: For each feature and each class, calculate the likelihood of observing a specific feature value given the class. This involves counting the occurrences of feature values for each class in the training data.

3. Calculate posterior probabilities: For a new, unseen data point, calculate the posterior probabilities for each class using Bayes’ theorem. The posterior probability of a class given the features is proportional to the prior probability of the class and the product of the likelihoods for each feature.

4. Make a prediction: As the projected class for the new data point, select the class with the highest posterior probability. This is the class that the algorithm believes is most likely given the observed features.

Example#

Let’s assume that we have the following dataset.

Step 1: Calculate prior probabilities

Based on the given data, we have the following prior probabilities:

• $P(\text{Disease} = \text{Influenza)} = 5/10$

• $P(\text{Disease} = \text{Common cold)} = 3/10$

• $P(\text{Disease} = \text{Other}) = 2/10$

Step 2: Calculate likelihoods

For each feature, we calculate the likelihood of observing “Yes” or “No” for each class as follows:

• $P(\text{Fever = Yes | Disease = Influenza}) = 3/5$

• $P(\text{Fever = Yes | Disease = Common cold}) = 1/3$

• $P(\text{Fever = Yes | Disease = Other}) = 1/2$

• $P(F\text{ever = No | Disease = Influenza}) = 2/5$

• $P(\text{Fever = No | Disease = Common cold}) = 2/3$

• $P(\text{Fever = No | Disease = Other}) = 1/2$

• $P(\text{Fatigue = Yes | Disease = Influenza}) = 1/5$

• $P(\text{Fatigue = Yes | Disease = Common cold}) = 2/3$

• $P(\text{Fatigue = Yes | Disease = Other}) = 1/2$

• $P(\text{Fatigue = No | Disease = Influenza}) = 4/5$ = 4/5

• $P(\text{Fatigue = No | Disease = Common cold}) = 1/3$

• $P(\text{Fatigue = No | Disease = Other}) = 1/2$

• $P(\text{Cough = Yes | Disease = Influenza}) = 2/5$

• $P(\text{Cough = Yes | Disease = Common cold}) = 2/3$

• $P(\text{Cough = Yes | Disease = Other}) = 1/2$

• $P(\text{Cough = No | Disease = Influenza}) = 3/5$

• $P(\text{Cough = No | Disease = Common cold}) = 1/3$

• $P(\text{Cough = No | Disease = Other}) = 1/2$

Step 3: Calculate posterior probabilities

Assume we have a new data point: $d = \text{{Fever = Yes, Fatigue = No, Cough = Yes}}.$

We need to calculate the following three probabilities:

• $P(\text{Disease = Influenza | Fever = Yes, Fatigue = No, Cough = Yes})$

• $P(\text{Disease = Common cold | Fever = Yes, Fatigue = No, Cough = Yes})$

• $P(\text{Disease = Other | Fever = Yes, Fatigue = No, Cough = Yes})$

Here is the posterior probability for $\text{Disease = Influenza}$:

$P(\text{Disease = Influenza | Fever = Yes, Fatigue = No, Cough = Yes})$ $= P(\text{Disease = Influenza})$ * $P(\text{Fever = Yes | Disease = Influenza)}$ * $P(\text{Fatigue = No | Disease = Influenza})$ * $P(\text{Cough = Yes | Disease = Influenza})$

$= 5/10 * 3/5 * 4/5 * 2/5 = 0.096.$

Here is the posterior probability for $\text{Disease = Common cold}$:

$P(\text{Disease = Common cold | Fever = Yes, Fatigue = No, Cough = Yes})$ $= P(\text{Disease = Common cold})$ * $P(\text{Fever = Yes | Disease = Common cold})$ * $P(\text{Fatigue = No | Disease = Common cold})$ * $P(\text{Cough = Yes | Disease = Common cold})$

$= 3/10 * 1/3 * 1/3 * 2/3 = 0.022.$

Here is the posterior probability for $\text{Disease = Other}$:

$P(\text{Disease = Other | Fever = Yes, Fatigue = No, Cough = Yes})$ $= P(\text{Disease = Other})$ * $P(\text{Fever = Yes | Disease = Other})$ * $P(\text{Fatigue = No | Disease = Other})$ * $P(\text{Cough = Yes | Disease = Other})$

$= 2/10 * 1/2 * 1/2 * 1/2 = 0.025.$

Step 4: Make a prediction

Because the posterior probability for $Disease = Influenza$ is maximum, we can assign the new data point d belonging to influenza.

Pros and cons of using Naïve Bayes#

Let’s now take a look at a few advantages and disadvantages of using the Naïve Bayes algorithm.

Advantages

Disadvantages

Conclusion and next steps#

This blog has provided a quick introduction to the Naïve Bayes algorithm. We started with a brief introduction to Bayes’ theorem, mentioned some use cases, and explored the advantages and disadvantages of using the Naïve Bayes algorithm for classification. We also demonstrated our example and showed each step.

However, your journey does not end here! To create models that are more reliable and accurate, you might want to experiment with various approaches and frameworks. We recommend that you look into the following courses offered by Educative: