Experiment Tracking

Experiment tracking

An experiment is a single run of the ML training pipeline. During ML modeling, the data scientist will have to tweak various aspects of the training process to achieve optimal model performance. The following are some of the things a data scientist can change:

• Data: Raw data may be sampled in different ways. Preprocessing and feature engineering may be done in various ways for various experiments. The train-test split fraction can also be changed.

• Model type: For example, the data scientist may decide to try gradient boosting instead of random forest.

• Hyperparameters: For a fixed model, hyperparameters may be changed.

• Others: Data based on a subset of features or loss function, for example.

As a concrete example, a data scientist starts training a new classifier. For the baseline, they use all the data, split 70:30 between train and test sets. They pick a support vector machine as the estimator and fix some of its hyperparameters. The first training run yields an accuracy of 80%. This is the baseline performance.

For the second run, the data scientist decides to change the classifier’s regularization from L2 to L1 but leave everything else the same. The second training run yields an accuracy of 85%, which they deem to be significantly different from the previous baseline. They now have a new baseline.

It’s important to keep track of each of these experiments so that even after a large number of them, you’ll be aware of the mapping for a particular experiment. A data scientist needs to know what changed in an experiment with respect to the previous run.

It's a good idea to change only one thing per training run so you can draw a causal relationship between the changed variable and the change in model performance.

Pipeline artifacts

An artifact is a file that the pipeline outputs. It can be a dataset (e.g., feature engineered data), a trained model, a diagnostic plot, or a run log. The following block diagram shows the artifacts that our pipeline outputs.