Experiment Tracking

Experiment tracking

An experiment is a single run of the ML training pipeline. During ML modeling, the data scientist will have to tweak various aspects of the training process to achieve optimal model performance. The following are some of the things a data scientist can change:

• Data: Raw data may be sampled in different ways. Preprocessing and feature engineering may be done in various ways for various experiments. The train-test split fraction can also be changed.

• Model type: For example, the data scientist may decide to try gradient boosting instead of random forest.

• Hyperparameters: For a fixed model, hyperparameters may be changed.

• Others: Data based on a subset of features or loss function, for example.

As a concrete example, a data scientist starts training a new classifier. For the baseline, they use all the data, split 70:30 between train and test sets. They pick a support vector machine as the estimator and fix some of its hyperparameters. The first training run yields an accuracy of 80%. This is the baseline performance.

For the second run, the data scientist decides to change the classifier’s regularization from L2 to L1 but leave everything else the same. The second training run yields an accuracy of 85%, which they deem to be significantly different from the previous baseline. ...