The following is a list of optimization steps and procedures that the genetic algorithm must follow to find the combination of hyperparameter values that produce the best ML model performance.

The steps for the genetic algorithm are as follows:

  1. Define the hyperparameters’ search space.

  2. Define the chromosome.

  3. Set the termination criteria.

  4. Initialize the population.

  5. Evaluate the fitness (performance).

  6. Select the chromosomes.

  7. Conduct the reproduction (crossover and mutation).

  8. Create a new generation.

  9. Iterate the process.

  10. Terminate the process.

1. Define the hyperparameters’ search space

The search space refers to the possible range of values that each defined hyperparameter can take at any given time. These hyperparameters mostly have integer values, float values, boolean values, or string values. It is essential to select a search space that is broad enough to incorporate potentially good combinations that can produce the best performance but not so broad that the search becomes inefficient.

Here are a few examples of hyperparameters from the KNN algorithm available on the scikit-learn library (KNeighborsClassifier()):

  • n_neighbors: This is the number of neighbors to use.

  • weights: This is the weight function used in prediction.

  • algorithm: This is the algorithm used to compute the nearest neighbors.

  • leaf_size: This is the leaf size passed to BallTree or KDTree.

2. Define chromosomes

The next step is to define chromosomes, which are different combinations of the hyperparameters’ values, forming the solution to the problem (classification or regression).

The chromosomes have genes, which refer to hyperparameters of the algorithm with specific values.

Some examples of chromosomes are listed below based on the selected hyperparameters from the KNeighborsClassifier():

Get hands-on with 1400+ tech skills courses.