Introduction: Test Set Analysis
Explore how to evaluate machine learning models using test set analysis, including ROC AUC and calibration visualizations. Understand how to translate model metrics into financial insights to demonstrate value to clients. Learn key considerations for delivering and monitoring models in real-world applications.
We'll cover the following...
Overview
This chapter presents several techniques for analyzing a model test set for deriving insights into likely model performance in the future. These techniques include the same model performance metrics we've already calculated, such as the ROC AUC, as well as new kinds of visualizations, such as the sloping of default risk by bins of predicted probability and the calibration of predicted probability.
After reading this chapter, you will be able to bridge ...