Introduction: Test Set Analysis

Overview

This chapter presents several techniques for analyzing a model test set for deriving insights into likely model performance in the future. These techniques include the same model performance metrics we've already calculated, such as the ROC AUC, as well as new kinds of visualizations, such as the sloping of default risk by bins of predicted probability and the calibration of predicted probability.

After reading this chapter, you will be able to bridge the gap between the theoretical metrics of machine learning and the financial metrics of the business world. You will be able to identify key insights while estimating the financial impact of a model and provide guidance to the client on how to realize this impact. We close with a discussion of the key elements to consider when delivering and deploying a model, such as the format of delivery and ways to monitor the model as it is being used.

Introduction to test set analysis

In the previous chapter, we used XGBoost to push model performance even higher than all our previous efforts and learned how to explain model predictions using SHAP values. Now, we will consider model building to be complete and address the remaining issues that need attention before delivering the model to the client. The key elements of this section are an analysis of the test set, including financial analysis, and things to consider when delivering a model to a client who wants to use it in the real world.

We look at the test set to understand how well the model will perform in the future. By calculating metrics we already know, like the ROC AUC, but now on the test set, we can gain confidence that our model will be useful for new data. We'll also learn some intuitive ways to visualize the power of the model for grouping customers into different levels of risk of default, such as a decile chart.

Your client will likely appreciate the efforts you made in creating a more accurate model or one with a higher ROC AUC. However, they will definitely appreciate understanding how much money the model can help them earn or save and will probably be happy to receive specific guidance on how to maximize the model's potential for this. A financial analysis of the test set can simulate different scenarios of model-based strategies and help the client pick one that works for them.

After completing the financial analysis, we will wrap up by discussing how to deliver a model for use by the client and how to monitor its performance over time.