Model Performance on the Test Set

Rigorous estimate of expected future performance

We already have some idea of the out-of-sample performance of the XGBoost model from the validation set. However, the validation set was used in model fitting via early stopping. The most rigorous estimate of expected future performance we can make should be created with data that was not used at all for model fitting. This was the reason for reserving a test dataset from the model-building process.

You may notice that we did examine the test set to some extent already, for example, in the chapter "Data Exploration and Cleaning," when assessing data quality and cleaning data. The gold standard for predictive modeling is to set aside a test set at the very beginning of a project and not examine it at all until the model is finished. This is the easiest way to ensure that none of the knowledge from the test set ...