Introduction

The following list outlines the optimization steps and procedures that the random search method must follow to find the combination of hyperparameter values that produce the best ML model performance:

  1. Define the hyperparameters for the ML model.

  2. Set the number of iterations for the random search.

  3. Create random combinations of hyperparameter values.

  4. Train the ML model using the randomly selected hyperparameters.

  5. Evaluate and record the performance of the ML model.

  6. Select the best-performing combination of hyperparameters.

1. Define the hyperparameters for the ML model

We start by specifying the range of values that each hyperparameter can take. In this step, we need to identify the hyperparameters of the ML algorithm we want to use and their function during the learning process. The easy way to do so is to read the documentation of the machine learning algorithm. For example, the logistic regression algorithm has about 15 parameters that can be combined. You can refer to the documentation provided by the scikit-learn library.

Here are some examples of hyperparameters from logistic regression:

  • penalty: This specifies the norm of the penalty.

  • C: This represents the inverse of regularization strength.

  • dual: This determines if the dual or primal formulation is implemented.

  • tol: This represents the tolerance for stopping criteria.

  • class_weight: This represents the weights associated with classes in the form {class_label: weight}.

  • solver: This represents the algorithm that optimizes the problem.

2. Set the number of iterations for the random search

In the second step, we have to define the total number of iterations to control the number of random hyperparameter combinations that will be evaluated. The default value is 10. The larger the number the better, because the random search method will be able to randomly try different combinations before selecting the best one.
However, it might take a lot of time to optimize because it might require more computational power if the model is complex and the dataset is large.

3. Create random combinations of hyperparameter values

For each iteration, the random search method randomly selects a value from the defined range of values for each hyperparameter. The total number of combinations of hyperparameter values depends on the number of iterations defined.

For example, if the number of iterations is 10, then the random search method will randomly create 10 different combinations of hyperparameter values.

4. Train the ML model using the randomly selected hyperparameters

In this step, the random search method will use the randomly selected hyperparameters to train the ML model on the training data. This process will be repeated for the rest of the combinations of the hyperparameter values.

Get hands-on with 1400+ tech skills courses.