Coding: Bayesian Optimization from Scratch

Bayesian optimization is a model-based method for finding the minimum of a function that is expensive to evaluate. It involves constructing a probabilistic model for the function and then exploiting this model to determine where to sample next.

General steps of Bayesian optimization

The general steps to implement Bayesian optimization are:

1. Specifying a surrogate model (usually a GP).

2. Defining an acquisition function based on this model.

3. Iterating the following steps for a number of rounds:

   1. Using the acquisition function to decide where to sample.

   2. Updating the surrogate model incorporating the new sample.

Example of machine learning

In the realm of practical ...