Gaussian Processes

What is a Gaussian process?

A Gaussian process (GP) is a probabilistic model used in machine learning for regression and classification tasks. It is a powerful tool for modeling complex, nonlinear functions, and it can be used to make predictions even when the underlying function is not fully understood.

A GP is a collection of random variables, any finite number of which have a joint Gaussian distribution. The main idea behind GPs is that a set of data points is generated by a process governed by a probability distribution.

GPs provide a way to specify a prior over functions, which can be updated using data to obtain a posterior distribution over functions. A mean function and a covariance function define this prior. The mean function can be set to zero, and the covariance function, also known as the kernel function, encodes the assumptions about the underlying function’s smoothness and periodicity.

One of the main advantages of GPs is that they provide a way to propagate uncertainty through the model. For example, given a set of observations, a GP can give a prediction of the function value at a new point, together with a measure of the uncertainty of the forecast. This uncertainty is usually represented by a variance, which quantifies the confidence of the prediction.

Gaussian process and Bayes’ theorem

GP is a probabilistic model closely linked to Bayes’ theorem. GPs are used for modeling a wide range of functions and can be used for regression and classification tasks.

In a GP, we assume that the underlying function we are trying to model is a sample from a multivariate normal distribution. We can then use Bayes’ theorem to infer the posterior distribution over the function given some observed data.

Mathematically, let $f(x)$ be the underlying function we want to model and $y$ be the observed data. This can be written as follows:

Here:

• $x$ is the input.

• ...