Search⌘ K
Home
Courses
Bayesian Machine Learning for Optimization in Python

Bayes’ Theorem in Machine Learning

Explore the use of Bayes theorem in machine learning to update probabilistic models for classification, regression, and clustering. Understand how to apply Bayesian statistics in Python to improve predictive accuracy and handle uncertainty through practical examples.

We'll cover the following...

One of the main ways Bayes’ theorem is used in machine learning is in developing probabilistic models that are used to make predictions about the likelihood of different outcomes based on data. These models typically involve the specification of a probability distribution, representing uncertainty about the outcome of interest. Bayes’ theorem can then be used to update the probability distribution based on new data or other evidence to refine the predictions made by the model.

Examples of Bayesian statistics in machine learning

Here are a few examples of the usage of Bayes’ theorem in machine learning.

Classification

Bayes’ theorem can be used to update the probability distribution over different classes based on the observed features of the data. For example, in a binary classification task, Bayes’ theorem can be used to update the probability that a given data point belongs to one of the two classes based on the observed features of the data point. This can improve the accuracy of classifiers by incorporating additional information about the data.

Here is an example of how Bayesian statistics can be incorporated into a classification task in Python:

Python 3.10.4
import scipy.stats as stats
import matplotlib.pyplot as plt
import numpy as np
# Define the prior probabilities for each class
prior_probs = [0.4, 0.6]
# Define the class-conditional probability distributions for each feature
feature_distributions = [
    stats.norm(loc=0, scale=1),  # Class 1
    stats.norm(loc=2, scale=1)   # Class 2
]
# Define the number of features
num_features = 2
# Define the number of classes
num_classes = 2
# Generate some synthetic data
data = np.random.randn(100, num_features)
data_y=np.random.randn(100)
# Compute the log-likelihoods for each class
log_likelihoods = np.zeros((num_classes, data.shape[0]))
for c in range(num_classes):
    for i in range(data.shape[0]):
        log_likelihoods[c, i] = np.sum([
            np.log(feature_distributions[c].pdf(data[i, j]))
            for j in range(num_features)
        ])
# Compute the posterior probabilities for each class
posterior_probs = np.zeros((num_classes, data.shape[0]))
for i in range(data.shape[0]):
    posterior_probs[:, i] = prior_probs * np.exp(log_likelihoods[:, i])
    posterior_probs[:, i] /= np.sum(posterior_probs[:, i])
# Classify the data points based on the posterior probabilities
predictions = np.argmax(posterior_probs, axis=0)
# Plot the data and the class-conditional probability distributions
x = np.linspace(-5, 5, 100)
fig, (ax1, ax2) = plt.subplots(2,1,sharex=True)
ax1.plot(x, feature_distributions[0].pdf(x), 'b-', lw=2, alpha=0.6, label='class 1')
ax1.plot(x, feature_distributions[1].pdf(x), 'r-', lw=2, alpha=0.6, label='class 2')
ax1.legend()
ax1.set_xlabel("Value of feature")
ax1.set_ylabel("Probability of feature")
ax1.set_title("Distribution of Feature in term of Classes")
ax2.scatter(data[:, 0], data[:, 1], c=predictions, cmap='bwr')
ax2.set_xlabel("Value of Feature 1")
ax2.set_ylabel("Value of Feature 2")
ax2.set_title("Distribution of Predicted Labels")
plt.tight_layout()
plt.show()
plt.savefig("output/classification.png")

In the code above, we generate some synthetic data with two features and two classes in lines 21–22. We also ...